IPython Parallel Push/Execute/Pull Demo

This notebook performs a very simple dot product of the same two random (square) matrices on all ipengines. It then gathers the results from all ipengines and confirms that all results are the same. This mostly demos how the push, execute, and pull pipeline works in the IPython Parallel client.

1. Allocate random matrices locally

In [113]:
import numpy as np
In [114]:
N = 2000
In [115]:
a = np.random.randn(N, N)
In [116]:
b = np.random.randn(N, N)

2. Compute their dot product locally

In [117]:
c = np.dot(a,b)

3. Initialize the IPython Parallel Client

In [118]:
from IPython.parallel import Client
In [119]:
cl = Client()
In [120]:
cl.ids
Out[120]:
[0, 1, 2]

4. Fetch a 'view' from the IPython parallel client

In [121]:
v = cl[:]

5. Import NumPy on all IPEngines

In [ ]:
v.execute('import numpy as np', block=True)

6. Push both random matrices to all IPEngines

In [122]:
v.push(dict(a=a, b=b), block=True)
Out[122]:
[None, None, None]

7. Compute dot product on all IPEngines

NOTE: We add 'print c' to the execute statement below in order to demo the display_outputs() function which shows the stdout, if any, of all remote engines as a result of executing the command. In general you do not need to print variables in order to interact with them.
In [123]:
r = v.execute('c = np.dot(a,b); print c', block=True)
In [124]:
r.display_outputs()
[stdout:0] 
[[  -4.67202217  104.39252912  -78.37783997 ...,  -29.30709274
     6.66403977   29.92526411]
 [  39.22517891   73.37959579 -140.66938625 ...,  -30.40772693
     9.21601775   29.80644042]
 [  31.60094895  -26.90207753   42.07903369 ...,    6.16564342  -40.9135798
   -57.82278587]
 ..., 
 [ -25.57059809  -63.64479316   22.76541772 ...,    1.68819694  -12.9543937
   -61.07450388]
 [  15.3646617    57.48457183  -34.15856279 ...,    7.34530517
   -16.26106107  -17.58811896]
 [ -13.26202943   75.42065094   74.86911918 ...,   -0.55862744
     6.72326022   10.38620045]]
[stdout:1] 
[[  -4.67202217  104.39252912  -78.37783997 ...,  -29.30709274
     6.66403977   29.92526411]
 [  39.22517891   73.37959579 -140.66938625 ...,  -30.40772693
     9.21601775   29.80644042]
 [  31.60094895  -26.90207753   42.07903369 ...,    6.16564342  -40.9135798
   -57.82278587]
 ..., 
 [ -25.57059809  -63.64479316   22.76541772 ...,    1.68819694  -12.9543937
   -61.07450388]
 [  15.3646617    57.48457183  -34.15856279 ...,    7.34530517
   -16.26106107  -17.58811896]
 [ -13.26202943   75.42065094   74.86911918 ...,   -0.55862744
     6.72326022   10.38620045]]
[stdout:2] 
[[  -4.67202217  104.39252912  -78.37783997 ...,  -29.30709274
     6.66403977   29.92526411]
 [  39.22517891   73.37959579 -140.66938625 ...,  -30.40772693
     9.21601775   29.80644042]
 [  31.60094895  -26.90207753   42.07903369 ...,    6.16564342  -40.9135798
   -57.82278587]
 ..., 
 [ -25.57059809  -63.64479316   22.76541772 ...,    1.68819694  -12.9543937
   -61.07450388]
 [  15.3646617    57.48457183  -34.15856279 ...,    7.34530517
   -16.26106107  -17.58811896]
 [ -13.26202943   75.42065094   74.86911918 ...,   -0.55862744
     6.72326022   10.38620045]]

8. Pull the results from all IPEngines

In [125]:
dot = v.pull('c', block=True)
In [126]:
print dot
[array([[  -4.67202217,  104.39252912,  -78.37783997, ...,  -29.30709274,
           6.66403977,   29.92526411],
       [  39.22517891,   73.37959579, -140.66938625, ...,  -30.40772693,
           9.21601775,   29.80644042],
       [  31.60094895,  -26.90207753,   42.07903369, ...,    6.16564342,
         -40.9135798 ,  -57.82278587],
       ..., 
       [ -25.57059809,  -63.64479316,   22.76541772, ...,    1.68819694,
         -12.9543937 ,  -61.07450388],
       [  15.3646617 ,   57.48457183,  -34.15856279, ...,    7.34530517,
         -16.26106107,  -17.58811896],
       [ -13.26202943,   75.42065094,   74.86911918, ...,   -0.55862744,
           6.72326022,   10.38620045]]), array([[  -4.67202217,  104.39252912,  -78.37783997, ...,  -29.30709274,
           6.66403977,   29.92526411],
       [  39.22517891,   73.37959579, -140.66938625, ...,  -30.40772693,
           9.21601775,   29.80644042],
       [  31.60094895,  -26.90207753,   42.07903369, ...,    6.16564342,
         -40.9135798 ,  -57.82278587],
       ..., 
       [ -25.57059809,  -63.64479316,   22.76541772, ...,    1.68819694,
         -12.9543937 ,  -61.07450388],
       [  15.3646617 ,   57.48457183,  -34.15856279, ...,    7.34530517,
         -16.26106107,  -17.58811896],
       [ -13.26202943,   75.42065094,   74.86911918, ...,   -0.55862744,
           6.72326022,   10.38620045]]), array([[  -4.67202217,  104.39252912,  -78.37783997, ...,  -29.30709274,
           6.66403977,   29.92526411],
       [  39.22517891,   73.37959579, -140.66938625, ...,  -30.40772693,
           9.21601775,   29.80644042],
       [  31.60094895,  -26.90207753,   42.07903369, ...,    6.16564342,
         -40.9135798 ,  -57.82278587],
       ..., 
       [ -25.57059809,  -63.64479316,   22.76541772, ...,    1.68819694,
         -12.9543937 ,  -61.07450388],
       [  15.3646617 ,   57.48457183,  -34.15856279, ...,    7.34530517,
         -16.26106107,  -17.58811896],
       [ -13.26202943,   75.42065094,   74.86911918, ...,   -0.55862744,
           6.72326022,   10.38620045]])]

9. Check remote results against local results

In [127]:
assert np.allclose(c, dot[0])
In [128]:
assert np.allclose(dot[0], dot[1])
In [129]:
assert np.allclose(dot[1], dot[2])