Sometimes you'll train on a smaller dataset that fits in memory, but need to predict or score for a much larger (possibly larger than memory) dataset. Perhaps your learning curve has leveled off, or you only have labels for a subset of the data.
In this situation, you can use ParallelPostFit to parallelize and distribute the scoring or prediction steps.
from dask.distributed import Client, progress # Scale up: connect to your own cluster with bmore resources # see http://dask.pydata.org/en/latest/setup.html client = Client(processes=False, threads_per_worker=4, n_workers=1, memory_limit='2GB') client
import numpy as np import dask.array as da from sklearn.datasets import make_classification
We'll generate a small random dataset with scikit-learn.
X_train, y_train = make_classification( n_features=2, n_redundant=0, n_informative=2, random_state=1, n_clusters_per_class=1, n_samples=1000) X_train[:5]
And we'll clone that dataset many times with
y_large represent our larger than memory dataset.
# Scale up: increase N, the number of times we replicate the data. N = 100 X_large = da.concatenate([da.from_array(X_train, chunks=X_train.shape) for _ in range(N)]) y_large = da.concatenate([da.from_array(y_train, chunks=y_train.shape) for _ in range(N)]) X_large
Since our training dataset fits in memory, we can use a scikit-learn estimator as the actual estimator fit during traning.
But we know that we'll want to predict for a large dataset, so we'll wrap the scikit-learn estimator with
from sklearn.linear_model import LogisticRegressionCV from dask_ml.wrappers import ParallelPostFit
clf = ParallelPostFit(LogisticRegressionCV(cv=3), scoring="r2")
See the note in the
dask-ml's documentation about when and why a
scoring parameter is needed: https://ml.dask.org/modules/generated/dask_ml.wrappers.ParallelPostFit.html#dask_ml.wrappers.ParallelPostFit.
Now we'll call
clf.fit. Dask-ML does nothing here, so this step can only use datasets that fit in memory.
Now that training is done, we'll turn to predicting for the full (larger than memory) dataset.
y_pred = clf.predict(X_large) y_pred
y_pred is Dask arary. Workers can write the predicted values to a shared file system, without ever having to collect the data on a single machine.
Or we can check the models score on the entire large dataset. The computation will be done in parallel, and no single machine will have to hold all the data.