This is one of the 100 recipes of the IPython Cookbook, the definitive guide to high-performance scientific computing and data science in Python.

4.9. Processing huge NumPy arrays with memory mapping¶

In [ ]:
import numpy as np


Writing a memory-mapped array¶

We create a memory-mapped array with a specific shape.

In [ ]:
nrows, ncols = 1000000, 100

In [ ]:
f = np.memmap('memmapped.dat', dtype=np.float32,
mode='w+', shape=(nrows, ncols))


Let's feed the array with random values, one column at a time because our system memory is limited!

In [ ]:
for i in range(ncols):
f[:,i] = np.random.rand(nrows)


We save the last column of the array.

In [ ]:
x = f[:,-1]


Now, we flush memory changes to disk by removing the object.

In [ ]:
del f


Reading a memory-mapped array from disk involves the same memmap function but with a different file mode. The data type and the shape need to be specified again, as this information is not stored in the file.

In [ ]:
f = np.memmap('memmapped.dat', dtype=np.float32, shape=(nrows, ncols))

In [ ]:
np.array_equal(f[:,-1], x)

In [ ]:
del f


You'll find all the explanations, figures, references, and much more in the book (to be released later this summer).

IPython Cookbook, by Cyrille Rossant, Packt Publishing, 2014 (500 pages).