This is one of the 100 recipes of the IPython Cookbook, the definitive guide to high-performance scientific computing and data science in Python.

5.10. Interacting with asynchronous parallel tasks in IPython

You need to start IPython engines (see previous recipe). The simplest option is to launch them from the Clusters tab in the notebook dashboard. In this recipe, we use four engines.

  1. Let's import a few modules.
In [ ]:
import time
import sys
from IPython import parallel
from IPython.display import clear_output, display
from IPython.html import widgets
  1. We create a Client.
In [ ]:
rc = parallel.Client()
  1. Now, we create a load balanced view on the IPython engines.
In [ ]:
view = rc.load_balanced_view()
  1. We define a simple function for our parallel tasks.
In [ ]:
def f(x):
    import time
    time.sleep(.1)
    return x*x
  1. We will run this function on 100 integer numbers in parallel.
In [ ]:
numbers = list(range(100))
  1. We execute f on our list numbers in parallel across all of our engines, using map_async(). This function returns immediately an AsyncResult object. This object allows us to retrieve interactively information about the tasks.
In [ ]:
ar = view.map_async(f, numbers)
  1. This object has a metadata attribute, a list of dictionaries for all engines. We can get the date of submission and completion, the status, the standard output and error, and other information.
In [ ]:
ar.metadata[0]
  1. Iterating over the AsyncResult instance works normally; the iteration progresses in real-time while the tasks are being completed.
In [ ]:
for _ in ar:
    print(_, end=', ')
  1. Now, we create a simple progress bar for our asynchronous tasks. The idea is to create a loop polling for the tasks' status at every second. An IntProgressWidget widget is updated in real-time and shows the progress of the tasks.
In [ ]:
def progress_bar(ar):
    # We create a progress bar.
    w = widgets.IntProgressWidget()
    # The maximum value is the number of tasks.
    w.max = len(ar.msg_ids)
    # We display the widget in the output area.
    display(w)
    # Repeat every second:
    while not ar.ready():
        # Update the widget's value with the
        # number of tasks that have finished
        # so far.
        w.value = ar.progress
        time.sleep(1)
    w.value = w.max
In [ ]:
ar = view.map_async(f, numbers)
In [ ]:
progress_bar(ar)
  1. Finally, it is easy to debug a parallel task on an engine. We can launch a Qt client on the remote kernel by calling %qtconsole within a %%px cell magic.
In [ ]:
%%px -t 0
%qtconsole

The Qt console allows us to inspect the remote namespace for debugging or analysis purposes.

You'll find all the explanations, figures, references, and much more in the book (to be released later this summer).

IPython Cookbook, by Cyrille Rossant, Packt Publishing, 2014 (500 pages).