How to recover best model from experiment log?

Due to system error or other reason where scan_object is no longer available, it's still possible to get best model/s using nothing but the experiment log. In the below notebook you will learn exactly how.

In [1]:
import sys
sys.path.insert(0, '/Users/mikko/Documents/GitHub/talos/')

import sys
sys.path.insert(0, '/Users/mikko/Documents/GitHub/wrangle/')

import talos
import wrangle
from keras.models import Sequential
from keras.layers import Dense
Using TensorFlow backend.

First we'll have to perform the Scan() experiment to produce the experiment log. Because the experiment log is stored on local machine, interrupted Scan() or other reason will not affect its availability. The experiment log is updated after each permutation; it contains an up-to-date record of the experiment.

In [2]:
# load the data
x, y = talos.templates.datasets.iris()
x_train, y_train, x_val, y_val = wrangle.array_split(x, y, .3)

# set the parameter space boundary
p = {'activation':['relu', 'elu'],
     'optimizer': ['Nadam', 'Adam'],
     'losses': ['logcosh'],
     'shapes': ['brick'],
     'first_neuron': [16, 32, 64, 128],
     'hidden_layers':[0, 1, 2, 3],
     'dropout': [.2, .3, .4],
     'batch_size': [20, 30, 40, 50],
     'epochs': [10]}

# define the input model
def iris_model(x_train, y_train, x_val, y_val, params):

    model = Sequential()
    model.add(Dense(params['first_neuron'], input_dim=4, activation=params['activation']))

    talos.utils.hidden_layers(model, params, 3)

    model.add(Dense(3, activation='softmax'))
    model.compile(optimizer=params['optimizer'], loss=params['losses'], metrics=['acc'])

    out = model.fit(x_train, y_train, callbacks=[talos.utils.ExperimentLogCallback('minimal_iris', params)],
                     batch_size=params['batch_size'],
                     epochs=params['epochs'],
                     validation_data=[x_val, y_val],
                     verbose=0)

    return out, model

# start the experiment
scan_object = talos.Scan(x=x_train,
                         y=y_train,
                         x_val=x_val,
                         y_val=y_val,
                         model=iris_model,
                         experiment_name='reactivate',
                         params=p,
                         round_limit=10)
100%|██████████| 10/10 [00:34<00:00,  3.28s/it]

Now we can assume the case where we no longer have access to the scan_object. In this Scan(...experiment_name...) was set to "reactivate" so we'll find a folder with that name in the present working directory. Next we have to find out what is the name of the experiment log.

In [3]:
# get the name of the experiment log
!ls -lhtr reactivate
total 96
-rw-r--r--  1 mikko  staff    16K Sep 26 20:17 092619200831.csv
-rw-r--r--  1 mikko  staff   1.3K Sep 26 20:18 092619201824.csv
-rw-r--r--  1 mikko  staff   1.3K Sep 26 22:13 092619221236.csv
-rw-r--r--  1 mikko  staff   1.3K Sep 26 22:18 092619221803.csv
-rw-r--r--  1 mikko  staff   1.3K Sep 26 22:31 092619223042.csv
-rw-r--r--  1 mikko  staff   1.3K Sep 26 22:35 092619223459.csv
-rw-r--r--  1 mikko  staff   1.3K Sep 26 22:35 092619223524.csv
-rw-r--r--  1 mikko  staff   1.3K Sep 26 22:56 092619225556.csv
-rw-r--r--  1 mikko  staff   1.3K Sep 26 23:04 092619230425.csv

In this case it will be the most recent one 092619223042.csv so let's go ahead and recover the best models.

In [5]:
from talos.utils.recover_best_model import recover_best_model
results, models = recover_best_model(x_train=x_train,
                                     y_train=y_train,
                                     x_val=x_val,
                                     y_val=y_val,
                                     experiment_log='reactivate/092619221803.csv',
                                     input_model=iris_model,
                                     n_models=5,
                                     task='multi_label')

Now we can access the cross-validation results:

In [6]:
results
Out[6]:
round_epochs val_loss val_acc loss acc activation batch_size dropout epochs first_neuron hidden_layers losses optimizer shapes crossval_mean_f1score
0 10 0.028760 0.955556 0.031489 0.885714 relu 20 0.3 10 128 2 logcosh Adam brick 0.930236
9 10 0.028902 0.933333 0.029437 0.866667 elu 50 0.3 10 128 2 logcosh Adam brick 0.900427
2 10 0.023247 0.888889 0.020156 0.942857 elu 30 0.3 10 128 2 logcosh Adam brick 0.980606
4 10 0.044717 0.866667 0.048760 0.780952 relu 30 0.4 10 128 2 logcosh Adam brick 0.529231
8 10 0.047718 0.866667 0.052877 0.761905 elu 30 0.3 10 64 1 logcosh Adam brick 0.860539

We can also access the models and make predictions with them:

In [9]:
models[0].predict(x_val)
Out[9]:
array([[4.0995691e-02, 7.6433158e-01, 1.9467270e-01],
       [5.5384491e-02, 8.0346000e-01, 1.4115542e-01],
       [1.4972138e-01, 7.5501865e-01, 9.5259979e-02],
       [1.1326348e-01, 7.1604651e-01, 1.7069000e-01],
       [3.7939314e-02, 6.4578909e-01, 3.1627160e-01],
       [7.1953669e-02, 7.3772454e-01, 1.9032180e-01],
       [1.9050457e-02, 5.2020442e-01, 4.6074522e-01],
       [3.7893724e-02, 6.0179770e-01, 3.6030859e-01],
       [9.7751850e-03, 3.3405378e-01, 6.5617102e-01],
       [1.1282027e-02, 3.9736888e-01, 5.9134912e-01],
       [3.4884610e-03, 2.8260693e-01, 7.1390456e-01],
       [1.4066804e-02, 4.4939899e-01, 5.3653419e-01],
       [9.8388308e-01, 1.5294192e-02, 8.2262064e-04],
       [9.4405776e-03, 2.6316279e-01, 7.2739667e-01],
       [5.3356937e-03, 2.5790238e-01, 7.3676193e-01],
       [6.6629532e-03, 2.9248431e-01, 7.0085275e-01],
       [1.0408297e-02, 2.8865287e-01, 7.0093888e-01],
       [9.2165405e-03, 4.4728053e-01, 5.4350299e-01],
       [6.1816044e-02, 7.1130496e-01, 2.2687899e-01],
       [5.8197163e-02, 7.0744330e-01, 2.3435953e-01],
       [9.7906607e-01, 1.9747239e-02, 1.1866244e-03],
       [4.9119215e-02, 6.6068804e-01, 2.9019269e-01],
       [9.7612959e-01, 2.2444952e-02, 1.4254292e-03],
       [4.5029860e-02, 6.6044801e-01, 2.9452211e-01],
       [9.9142039e-01, 8.2095861e-03, 3.7001175e-04],
       [9.7515500e-01, 2.3289582e-02, 1.5554430e-03],
       [5.3465478e-02, 6.7216325e-01, 2.7437130e-01],
       [3.2891510e-03, 1.4500402e-01, 8.5170686e-01],
       [4.0943369e-02, 7.4083865e-01, 2.1821795e-01],
       [9.7216946e-01, 2.5830602e-02, 1.9999193e-03],
       [5.3125862e-02, 7.0550537e-01, 2.4136871e-01],
       [1.1228154e-01, 7.2705197e-01, 1.6066651e-01],
       [2.1662652e-02, 5.3180271e-01, 4.4653463e-01],
       [4.6057135e-02, 7.8316230e-01, 1.7078057e-01],
       [3.8668580e-02, 5.8559459e-01, 3.7573683e-01],
       [2.0084916e-02, 4.8307988e-01, 4.9683511e-01],
       [2.2133207e-02, 5.7616937e-01, 4.0169743e-01],
       [9.6933258e-01, 2.8047977e-02, 2.6194439e-03],
       [6.2396564e-02, 6.9619435e-01, 2.4140903e-01],
       [9.8845774e-01, 1.0924198e-02, 6.1809190e-04],
       [1.1867868e-02, 2.8613701e-01, 7.0199513e-01],
       [1.3917084e-02, 4.5086253e-01, 5.3522038e-01],
       [3.7958041e-02, 7.3275220e-01, 2.2928977e-01],
       [1.5825152e-02, 5.4144788e-01, 4.4272691e-01],
       [9.7893941e-01, 1.9864958e-02, 1.1956602e-03]], dtype=float32)