Is there a way to run multiple cells simultaneously in IPython notebook?

This does not answer your question directly but I think it would help a lot of people that are having the same problem. You can move variables between notebooks easily and then continue running the functions on another notebook then move the result back to the main notebook.

For example:

Notebook 1:

%store X
%store y

Notebook 2:

%store -r X
%store -r y

new_df = ...
%store new_df

Notebook 1:

%store -r new_df 

I got very hopeful with Matt answer of the ipp module, but the truth is that the ipp does not run two cells in pararell. Ipp lets you work in two or more engines but not simultaneously.

Take this example, you run the first code and 1 second later you run the second code, each code in different cells:

%%px --targets 0
import time
for i in range(0,6):
    time.sleep(1)
    print(time.ctime())

Gives:

Thu Jun 16 10:30:53 2022
Thu Jun 16 10:30:54 2022
Thu Jun 16 10:30:55 2022
Thu Jun 16 10:30:56 2022
Thu Jun 16 10:30:57 2022

And

%%px --targets 1
import time
for i in range(0,6):
    time.sleep(1)
    print(time.ctime())

Gives:

Thu Jun 16 10:30:59 2022
Thu Jun 16 10:31:00 2022
Thu Jun 16 10:31:01 2022
Thu Jun 16 10:31:02 2022
Thu Jun 16 10:31:03 2022

So in conclusion, the cells are not running at the same time, they are just running in different engines. The second cell waits the 1st one to finish, and once it finishes the second cell starts.

Hope there is simple solution for this -.-

PD: Here is the image Code in jupyter notebook


Yes. Here is the documentation for ipyparallel (formerly IPython parallel) that will show you how to spawn multiple IPython kernel. After you are free to distribute the work across cores, and you can prefix cells with %%px0 %%px1... %%px999 (once set up) to execute a cell on a specific engine, which in practice correspond to parallel execution of cell. I would suggest having a look at Dask as well.