Using %matplotlib notebook after %matplotlib inline in Jupyter Notebook doesn't work

You just have the wrong order of your commands. A backend should be set before importing pyplot in jupyter. Or in other words, after changing the backend, pyplot needs to be imported again.

Therefore call %matplotlib ... prior to importing pyplot.

In first cell:

%matplotlib inline
import matplotlib.pyplot as plt
plt.plot([1,1.6,3])

In second cell:

%matplotlib notebook
#calling it a second time may prevent some graphics errors
%matplotlib notebook  
import matplotlib.pyplot as plt
plt.plot([1,1.6,3])

Edit: turns out that you can in fact change backends dynamically on jupyter. Still leaving the answer here because I think it's relevant and explains some matplotlib magic that can pop out sometimes.

The magic command, as seen in the source code, is calling matplotlib.pyplot.switch_backend(newbackend) to change the backend. As stated in matplotlib's docs:

matplotlib.pyplot.switch_backend(newbackend)

Switch the default backend. This feature is experimental, and is only expected to work switching to an image backend. e.g., if you have a bunch of PostScript scripts that you want to run from an interactive ipython session, you may want to switch to the PS backend before running them to avoid having a bunch of GUI windows popup. If you try to interactively switch from one GUI backend to another, you will explode..

So you really have to restart the kernel each time you switch backends, because matplotlib has a problem to switch the backend after being used.

This problem is mainly due to incompatibilities between different main-loops of the GUI backend. Because normally each backend is also taking care of threads and user input you can't run Qt and Tkinter side-by-side. So that limitation is carried over to jupyter.

Also see this question: How to switch backends in matplotlib / Python