This notebook shows some more advanced features of skorch
. More examples will be added with time.
import torch
from torch import nn
import torch.nn.functional as F
torch.manual_seed(0);
We load a toy classification task from sklearn
.
import numpy as np
from sklearn.datasets import make_classification
X, y = make_classification(1000, 20, n_informative=10, random_state=0)
X = X.astype(np.float32)
X.shape, y.shape, y.mean()
((1000, 20), (1000,), 0.5)
pytorch
classification module
¶We define a vanilla neural network with two hidden layers. The output layer should have 2 output units since there are two classes. In addition, it should have a softmax nonlinearity, because later, when calling predict_proba
, the output from the forward
call will be used.
from skorch.net import NeuralNetClassifier
class ClassifierModule(nn.Module):
def __init__(
self,
num_units=10,
nonlin=F.relu,
dropout=0.5,
):
super(ClassifierModule, self).__init__()
self.num_units = num_units
self.nonlin = nonlin
self.dropout = dropout
self.dense0 = nn.Linear(20, num_units)
self.nonlin = nonlin
self.dropout = nn.Dropout(dropout)
self.dense1 = nn.Linear(num_units, 10)
self.output = nn.Linear(10, 2)
def forward(self, X, **kwargs):
X = self.nonlin(self.dense0(X))
X = self.dropout(X)
X = F.relu(self.dense1(X))
X = F.softmax(self.output(X), dim=-1)
return X
Callbacks are a powerful and flexible way to customize the behavior of your neural network. They are all called at specific points during the model training, e.g. when training starts, or after each batch. Have a look at the skorch.callbacks
module to see the callbacks that are already implemented.
Although skorch
comes with a handful of useful callbacks, you may find that you would like to write your own callbacks. Doing so is straightforward, just remember these rules:
skorch.callbacks.Callback
.on_
-methods provided by the parent class (e.g. on_batch_begin
or on_epoch_end
).on_
-methods first get the NeuralNet
instance, and, where appropriate, the local data (e.g. the data from the current batch). The method should also have **kwargs
in the signature for potentially unused arguments.initialize
method.Here is an example of a callback that remembers at which epoch the validation accuracy reached a certain value. Then, when training is finished, it calls a mock Twitter API and tweets that epoch. We proceed as follows:
__init__
.initialize
.from skorch.callbacks import Callback
def tweet(msg):
print("~" * 60)
print("*tweet*", msg, "#skorch #pytorch")
print("~" * 60)
class AccuracyTweet(Callback):
def __init__(self, min_accuracy):
self.min_accuracy = min_accuracy
def initialize(self):
self.critical_epoch_ = -1
def on_epoch_end(self, net, **kwargs):
if self.critical_epoch_ > -1:
return
# look at the validation accuracy of the last epoch
if net.history[-1, 'valid_acc'] >= self.min_accuracy:
self.critical_epoch_ = len(net.history)
def on_train_end(self, net, **kwargs):
if self.critical_epoch_ < 0:
msg = "Accuracy never reached {} :(".format(self.min_accuracy)
else:
msg = "Accuracy reached {} at epoch {}!!!".format(
self.min_accuracy, self.critical_epoch_)
tweet(msg)
Now we initialize a NeuralNetClassifier
and pass your new callback in a list to the callbacks
argument. After that, we train the model and see what happens.
net = NeuralNetClassifier(
ClassifierModule,
max_epochs=10,
lr=0.02,
warm_start=True,
callbacks=[AccuracyTweet(min_accuracy=0.7)],
)
net.fit(X, y)
epoch train_loss valid_acc valid_loss dur ------- ------------ ----------- ------------ ------ 1 0.6908 0.5950 0.6842 0.0315 2 0.6876 0.5950 0.6815 0.0256 3 0.6853 0.6100 0.6789 0.0365 4 0.6882 0.5950 0.6769 0.0290 5 0.6780 0.6000 0.6743 0.0230 6 0.6730 0.6100 0.6717 0.0238 7 0.6664 0.6150 0.6698 0.0232 8 0.6670 0.6100 0.6670 0.0234 9 0.6667 0.6300 0.6646 0.0342 10 0.6624 0.6350 0.6624 0.0234 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *tweet* Accuracy never reached 0.7 :( #skorch #pytorch ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<class 'skorch.net.NeuralNetClassifier'>[initialized]( module_=ClassifierModule( (dense0): Linear(in_features=20, out_features=10, bias=True) (dropout): Dropout(p=0.5) (dense1): Linear(in_features=10, out_features=10, bias=True) (output): Linear(in_features=10, out_features=2, bias=True) ), )
Oh no, our model never reached a validation accuracy of 0.7. Let's train some more (this is possible because we set warm_start=True
):
net.fit(X, y)
11 0.6647 0.6500 0.6598 0.0240 12 0.6573 0.6650 0.6575 0.0240 13 0.6458 0.6700 0.6549 0.0250 14 0.6528 0.6750 0.6525 0.0242 15 0.6476 0.6700 0.6502 0.0240 16 0.6483 0.6750 0.6476 0.0244 17 0.6514 0.6800 0.6452 0.0254 18 0.6365 0.6850 0.6422 0.0231 19 0.6335 0.7000 0.6390 0.0230 20 0.6381 0.7100 0.6363 0.0233 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *tweet* Accuracy reached 0.7 at epoch 19!!! #skorch #pytorch ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<class 'skorch.net.NeuralNetClassifier'>[initialized]( module_=ClassifierModule( (dense0): Linear(in_features=20, out_features=10, bias=True) (dropout): Dropout(p=0.5) (dense1): Linear(in_features=10, out_features=10, bias=True) (output): Linear(in_features=10, out_features=2, bias=True) ), )
Finally, the validation score exceeded 0.7. Hooray!
Say you would like to use a learning rate schedule with your neural net, but you don't know what parameters are best for that schedule. Wouldn't it be nice if you could find those parameters with a grid search? With skorch
, this is possible. Below, we show how to access the parameters of your callbacks.
To simplify the access to your callback parameters, it is best if you give your callback a name. This is achieved by passing the callbacks
parameter a list of name, callback tuples, such as:
callbacks=[
('scheduler', LearningRateScheduler)),
...
],
This way, you can access your callbacks using the double underscore semantics (as, for instance, in an sklearn
Pipeline
):
callbacks__scheduler__epoch=50,
So if you would like to perform a grid search on, say, the number of units in the hidden layer and the learning rate schedule, it could look something like this:
param_grid = {
'module__num_units': [50, 100, 150],
'callbacks__scheduler__epoch': [10, 50, 100],
}
Note: If you would like to refresh your knowledge on grid search, look here, here, or in the Basic_Usage notebok.
Below, we show how accessing the callback parameters works our AccuracyTweet
callback:
net = NeuralNetClassifier(
ClassifierModule,
max_epochs=10,
lr=0.1,
warm_start=True,
callbacks=[
('tweet', AccuracyTweet(min_accuracy=0.7)),
],
callbacks__tweet__min_accuracy=0.6,
)
net.fit(X, y)
epoch train_loss valid_acc valid_loss dur ------- ------------ ----------- ------------ ------ 1 0.7139 0.5500 0.6933 0.0317 2 0.6916 0.5950 0.6873 0.0283 3 0.6829 0.5800 0.6814 0.0212 4 0.6671 0.6050 0.6718 0.0228 5 0.6670 0.6200 0.6639 0.0229 6 0.6622 0.6350 0.6546 0.0236 7 0.6371 0.6550 0.6429 0.0240 8 0.6293 0.6700 0.6312 0.0236 9 0.6170 0.6650 0.6200 0.0240 10 0.6204 0.6750 0.6119 0.0255 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *tweet* Accuracy reached 0.6 at epoch 4!!! #skorch #pytorch ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<class 'skorch.net.NeuralNetClassifier'>[initialized]( module_=ClassifierModule( (dense0): Linear(in_features=20, out_features=10, bias=True) (dropout): Dropout(p=0.5) (dense1): Linear(in_features=10, out_features=10, bias=True) (output): Linear(in_features=10, out_features=2, bias=True) ), )
As you can see, by passing callbacks__tweet__min_accuracy=0.6
, we changed that parameter. The same can be achieved by calling the set_params
method with the corresponding arguments:
net.set_params(callbacks__tweet__min_accuracy=0.75)
<class 'skorch.net.NeuralNetClassifier'>[initialized]( module_=ClassifierModule( (dense0): Linear(in_features=20, out_features=10, bias=True) (dropout): Dropout(p=0.5) (dense1): Linear(in_features=10, out_features=10, bias=True) (output): Linear(in_features=10, out_features=2, bias=True) ), )
net.fit(X, y)
epoch train_loss valid_acc valid_loss dur ------- ------------ ----------- ------------ ------ 11 0.5845 0.7000 0.6016 0.0237 12 0.5831 0.7050 0.5915 0.0221 13 0.5854 0.7200 0.5788 0.0221 14 0.5582 0.7150 0.5729 0.0226 15 0.5601 0.7150 0.5692 0.0247 16 0.5468 0.7250 0.5662 0.0222 17 0.5333 0.7300 0.5583 0.0230 18 0.5592 0.7200 0.5555 0.0240 19 0.5295 0.7300 0.5488 0.0238 20 0.5232 0.7300 0.5428 0.0239 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *tweet* Accuracy never reached 0.75 :( #skorch #pytorch ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<class 'skorch.net.NeuralNetClassifier'>[initialized]( module_=ClassifierModule( (dense0): Linear(in_features=20, out_features=10, bias=True) (dropout): Dropout(p=0.5) (dense1): Linear(in_features=10, out_features=10, bias=True) (output): Linear(in_features=10, out_features=2, bias=True) ), )
Dataset
s¶We encourage you to not pass Dataset
s to net.fit
but to let skorch handle Dataset
s internally. Nonetheless, there are situations where passing Dataset
s to net.fit
is hard to avoid (e.g. if you want to load the data lazily during the training). This is supported by skorch but may have some unwanted side-effects relating to sklearn. For instance, Dataset
s cannot split into train and validation in a stratified fashion without explicit knowledge of the classification targets.
Below we show what happens when you try to fit with Dataset
and the stratified split fails:
class MyDataset(torch.utils.data.Dataset):
"""Dataset with inaccessible X and y"""
def __init__(self, X, y):
self.xx = X
self.yy = y
assert len(X) == len(y)
def __len__(self):
return len(self.xx)
def __getitem__(self, i):
return self.xx[i], self.yy[i]
X, y = make_classification(1000, 20, n_informative=10, random_state=0)
X = X.astype(np.float32)
dataset = MyDataset(X, y)
net = NeuralNetClassifier(ClassifierModule)
try:
net.fit(dataset, y=None)
except ValueError as e:
print("Error:", e)
Error: Stratified CV requires explicitely passing a suitable y.
net.train_split.stratified
True
As you can see, the stratified split fails since y
is not known. There are two solutions to this:
net.train_split.stratified=False
)y
explicitly (if possible), even if it is implicitely contained in the Dataset
The second solution is shown below:
net.fit(dataset, y=y)
Re-initializing module! epoch train_loss valid_acc valid_loss dur ------- ------------ ----------- ------------ ------ 1 0.7097 0.5400 0.6932 0.0099 2 0.7087 0.5500 0.6913 0.0088 3 0.7182 0.5550 0.6895 0.0088 4 0.7088 0.5550 0.6881 0.0144 5 0.7097 0.5550 0.6867 0.0089 6 0.6943 0.5550 0.6854 0.0085 7 0.6965 0.5700 0.6842 0.0090 8 0.6975 0.5750 0.6830 0.0087 9 0.6896 0.5950 0.6819 0.0084 10 0.6950 0.6000 0.6809 0.0095
<class 'skorch.net.NeuralNetClassifier'>[initialized]( module_=ClassifierModule( (dense0): Linear(in_features=20, out_features=10, bias=True) (dropout): Dropout(p=0.5) (dense1): Linear(in_features=10, out_features=10, bias=True) (output): Linear(in_features=10, out_features=2, bias=True) ), )
skorch has built-in support for dictionaries as data containers. Here we show a somewhat contrived example of how to use dicts, but it should get the point across. First we create data and put it into a dictionary X_dict
with two keys X0
and X1
:
X, y = make_classification(1000, 20, n_informative=10, random_state=0)
X0, X1 = X[:, :10].astype(np.float32), X[:, 10:].astype(np.float32)
X_dict = {'X0': X0, 'X1': X1}
When skorch passes the dict to the pytorch module, it will pass the data as keyword arguments to the forward call. That means that we should accept the two keys XO
and X1
in the forward method, as shown below:
class ClassifierWithDict(nn.Module):
def __init__(
self,
num_units0=50,
num_units1=50,
nonlin=F.relu,
dropout=0.5,
):
super(ClassifierWithDict, self).__init__()
self.num_units0 = num_units0
self.num_units1 = num_units1
self.nonlin = nonlin
self.dropout = dropout
self.dense0 = nn.Linear(10, num_units0)
self.dense1 = nn.Linear(10, num_units1)
self.nonlin = nonlin
self.dropout = nn.Dropout(dropout)
self.output = nn.Linear(num_units0 + num_units1, 2)
# NOTE: We accept X0 and X1, the keys from the dict, as arguments
def forward(self, X0, X1, **kwargs):
X0 = self.nonlin(self.dense0(X0))
X0 = self.dropout(X0)
X1 = self.nonlin(self.dense1(X1))
X1 = self.dropout(X1)
X = torch.cat((X0, X1), dim=1)
X = F.relu(X)
X = F.softmax(self.output(X), dim=-1)
return X
As long as we keep this in mind, we are good to go.
net = NeuralNetClassifier(ClassifierWithDict, verbose=0)
net.fit(X_dict, y)
<class 'skorch.net.NeuralNetClassifier'>[initialized]( module_=ClassifierWithDict( (dense0): Linear(in_features=10, out_features=50, bias=True) (dense1): Linear(in_features=10, out_features=50, bias=True) (dropout): Dropout(p=0.5) (output): Linear(in_features=100, out_features=2, bias=True) ), )
FunctionTransformer
and GridSearch
¶from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer
from sklearn.model_selection import GridSearchCV
sklearn makes the assumption that incoming data should be numpy/sparse arrays or something similar. This clashes with the use of dictionaries. Unfortunately, it is sometimes impossible to work around that for now (for instance using skorch with BaggingClassifier
). Other times, there are possibilities.
When we have a preprocessing pipeline that involves FunctionTransformer
, we have to pass the parameter validate=False
so that sklearn allows the dictionary to pass through:
pipe = Pipeline([
('do-nothing', FunctionTransformer(validate=False)),
('net', net),
])
pipe.fit(X_dict, y)
Pipeline(memory=None, steps=[('do-nothing', FunctionTransformer(accept_sparse=False, func=None, inv_kw_args=None, inverse_func=None, kw_args=None, pass_y='deprecated', validate=False)), ('net', <class 'skorch.net.NeuralNetClassifier'>[initialized]( module_=ClassifierWithDict( (dense0): Linear(in... (dropout): Dropout(p=0.5) (output): Linear(in_features=100, out_features=2, bias=True) ), ))])
When trying a grid or randomized search, it is not that easy to pass a dict. If we try, we will get an error:
param_grid = {
'net__module__num_units0': [10, 25, 50],
'net__module__num_units1': [10, 25, 50],
'net__lr': [0.01, 0.1],
}
grid_search = GridSearchCV(pipe, param_grid, scoring='accuracy', verbose=1)
try:
grid_search.fit(X_dict, y)
except Exception as e:
print(e)
Found input variables with inconsistent numbers of samples: [2, 1000]
The error above occurs because sklearn gets the length of the input data, which is 2 for the dict, and believes that is inconsistent with the length of the target (1000).
To get around that, skorch provides a helper class called SliceDict
. It allows us to wrap our dictionaries so that they also behave like a numpy array:
from skorch.helper import SliceDict
X_slice_dict = SliceDict(X0=X0, X1=X1) # X_slice_dict = SliceDict(**X_dict) would also work
The SliceDict shows the correct length, shape, and is sliceable across values:
print("Length of dict: {}, length of SliceDict: {}".format(len(X_dict), len(X_slice_dict)))
print("Shape of SliceDict: {}".format(X_slice_dict.shape))
Length of dict: 2, length of SliceDict: 1000 Shape of SliceDict: (1000,)
print("Slicing the SliceDict slices across values: {}".format(X_slice_dict[:2]))
Slicing the SliceDict slices across values: SliceDict(**{'X0': array([[-0.96583462, -2.18907046, 0.16985609, 0.81384557, -3.37520909, -2.14305973, -0.39585084, 2.94195771, -2.19106054, 1.24439669], [-0.45476699, 4.33976793, -0.48572844, -4.8843298 , -2.8836503 , 2.60972047, -1.95287597, -0.09192174, 0.07970932, -0.08938338]], dtype=float32), 'X1': array([[ 0.04351204, -0.51509613, -0.86073655, -1.10971689, 0.31839254, -0.82319731, -1.05630398, -0.89645284, 0.37592441, -1.08496511], [-0.60726726, -1.06743085, 0.48804346, -0.50230557, 0.55743027, 1.01592004, -1.99535823, 2.90304255, -0.97392982, 2.17533231]], dtype=float32)})
With this, we can call GridSearch
just as expected:
grid_search.fit(X_slice_dict, y)
Fitting 3 folds for each of 18 candidates, totalling 54 fits
[Parallel(n_jobs=1)]: Done 54 out of 54 | elapsed: 15.6s finished
GridSearchCV(cv=None, error_score='raise', estimator=Pipeline(memory=None, steps=[('do-nothing', FunctionTransformer(accept_sparse=False, func=None, inv_kw_args=None, inverse_func=None, kw_args=None, pass_y='deprecated', validate=False)), ('net', <class 'skorch.net.NeuralNetClassifier'>[initialized]( module_=ClassifierWithDict( (dense0): Linear(in... (dropout): Dropout(p=0.5) (output): Linear(in_features=100, out_features=2, bias=True) ), ))]), fit_params=None, iid=True, n_jobs=1, param_grid={'net__module__num_units0': [10, 25, 50], 'net__module__num_units1': [10, 25, 50], 'net__lr': [0.01, 0.1]}, pre_dispatch='2*n_jobs', refit=True, return_train_score='warn', scoring='accuracy', verbose=1)
grid_search.best_score_, grid_search.best_params_
(0.76400000000000001, {'net__lr': 0.1, 'net__module__num_units0': 50, 'net__module__num_units1': 50})