Rewrite dlnd_project2 in kur

In [1]:
!kur --version
Kur, by Deepgram -- deep learning made easy
Version: 0.3.0
Homepage: https://kur.deepgram.com
In [12]:
%%writefile dlnd_p2.yml

---
settings:

  # Where to get the data
  cifar: &cifar
    url: "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
    checksum: "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce"
    path: "~/kur"           
                        # I am tempting to read supplier source code to understand better how to use 
                        # each supplier (I see a few available, csv, pickle), do you think 
                        # it is a good idea to spend time on?  #################################################

                        
  # Backend to use                                      
  backend:
    name: keras         # how many backend does kur have, and what different do they make? ##################### 
                        # some complaint about keras performance, do you think it is an issue? #################

  # Hyperparameters
  cnn:
    kernels: 20
    size: [5, 5]
    strides: [2, 2]                

  pool:
    size: [2,2]
    strides: [2,2]
    type: "max"                         

model:
  - input: images                       
  - convolution:                        
      kernels: "{{cnn.kernels}}"
      size: "{{cnn.size}}"
      strides: "{{ cnn.strides }}"
  - activation: relu
  - pool:
      size: "{{pool.size}}"
      strides: "{{pool.strides}}"
      type: "{{pool.type}}"             
  - flatten:
  - dense: 15                           
  - dense: 10
  - activation: softmax
    name: labels

train:
  data:
    - cifar:
        <<: *cifar
        parts: [1]                      # only use dataset part 1 to train
  provider:
    batch_size: 128
    num_batches: 1000                   # the entire part 1 will be used
  log: t3/cifar-log                   
  epochs: 20
  weights:
    initial: t3/cifar.best.valid.w    # do you think is it essential to implement weight visualization? #############
    best: t3/cifar.best.train.w       # visualizing weights on different layers is used in explaining cnn on image###
    last: t3/cifar.last.w             # or is it not important for help experimenting on models #####################

  optimizer:
    name: adam
    learning_rate: 0.001

validate:
  data:
    - cifar:
       <<: *cifar
       parts: 5                          # only use dataset part 5 as validation set
  provider:
    num_batches: 50                      # the project 2 only used 5000 data points as validation set
  weights: t3/cifar.best.valid.w

test: &test
  data:
    - cifar:
       <<: *cifar
       parts: test
  weights: t3/cifar.best.valid.w
  provider:
    num_batches: 1000                     # the entire part test will be used

evaluate:
  <<: *test
  destination: t3/cifar.results.pkl

loss:
  - target: labels                        # in the project: training loss and valid_accuracy are printed #############
    name: categorical_crossentropy        # this should be a matter of personal taste, won't really affect anything##
...
Overwriting dlnd_p2.yml
In [13]:
%pwd
Out[13]:
'/Users/Natsume/Downloads/kur_road'
In [14]:
%ls
Kur_Road.ipynb       dlnd_p2_dropout.yml  t1/
cifar-log/           dlnd_p2_kur.ipynb    t2/
dlnd_p2.yml          kur/                 t3/
In [15]:
!kur -v train dlnd_p2.yml
[INFO 2017-02-28 22:28:48,972 kur.kurfile:638] Parsing source: dlnd_p2.yml, included by top-level.
[INFO 2017-02-28 22:28:48,989 kur.kurfile:79] Parsing Kurfile...
[INFO 2017-02-28 22:28:49,038 kur.loggers.binary_logger:87] Log does not exist. Creating path: t3/cifar-log
[INFO 2017-02-28 22:28:56,014 kur.providers.batch_provider:54] Batch size set to: 128
[INFO 2017-02-28 22:28:56,014 kur.providers.batch_provider:60] Maximum number of batches set to: 1000
[INFO 2017-02-28 22:29:02,413 kur.providers.batch_provider:54] Batch size set to: 32
[INFO 2017-02-28 22:29:02,413 kur.providers.batch_provider:60] Maximum number of batches set to: 50
[INFO 2017-02-28 22:29:02,414 kur.backend.backend:80] Creating backend: keras
[INFO 2017-02-28 22:29:02,414 kur.backend.backend:83] Backend variants: none
[INFO 2017-02-28 22:29:02,414 kur.backend.keras_backend:122] No particular backend for Keras has been requested.
[INFO 2017-02-28 22:29:03,578 kur.backend.keras_backend:191] Keras is loaded. The backend is: theano
[INFO 2017-02-28 22:29:03,579 kur.model.model:260] Enumerating the model containers.
[INFO 2017-02-28 22:29:03,579 kur.model.model:265] Assembling the model dependency graph.
[INFO 2017-02-28 22:29:03,579 kur.model.model:280] Connecting the model graph.
[INFO 2017-02-28 22:29:04,579 kur.model.model:284] Model inputs:  images
[INFO 2017-02-28 22:29:04,579 kur.model.model:285] Model outputs: labels
[INFO 2017-02-28 22:29:04,580 kur.kurfile:310] Ignoring missing initial weights: t3/cifar.best.valid.w. If this is undesireable, set "must_exist" to "yes" in the approriate "weights" section.
[INFO 2017-02-28 22:29:05,910 kur.providers.batch_provider:54] Batch size set to: 2
[INFO 2017-02-28 22:29:05,910 kur.providers.batch_provider:60] Maximum number of batches set to: 1
[INFO 2017-02-28 22:29:05,913 kur.backend.keras_backend:654] Waiting for model to finish compiling...
[INFO 2017-02-28 22:29:05,956 kur.model.executor:256] No historical training loss available from logs.
[INFO 2017-02-28 22:29:05,956 kur.model.executor:264] No historical validation loss available from logs.
[INFO 2017-02-28 22:29:05,956 kur.model.executor:270] No previous epochs.

Epoch 1/20, loss=2.035: 100%|██████| 10000/10000 [00:02<00:00, 3344.74samples/s]
[INFO 2017-02-28 22:29:08,954 kur.model.executor:390] Training loss: 2.035
[INFO 2017-02-28 22:29:08,954 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
[INFO 2017-02-28 22:29:09,202 kur.providers.batch_provider:54] Batch size set to: 2
[INFO 2017-02-28 22:29:09,203 kur.providers.batch_provider:60] Maximum number of batches set to: 1
[INFO 2017-02-28 22:29:09,208 kur.backend.keras_backend:654] Waiting for model to finish compiling...
Validating, loss=1.854: 100%|████████| 1600/1600 [00:00<00:00, 4341.85samples/s]
[INFO 2017-02-28 22:29:09,585 kur.model.executor:175] Validation loss: 1.854
[INFO 2017-02-28 22:29:09,585 kur.model.executor:422] Saving best historical validation weights: t3/cifar.best.valid.w

Epoch 2/20, loss=1.693: 100%|██████| 10000/10000 [00:02<00:00, 4339.70samples/s]
[INFO 2017-02-28 22:29:11,897 kur.model.executor:390] Training loss: 1.693
[INFO 2017-02-28 22:29:11,897 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.641: 100%|████████| 1600/1600 [00:00<00:00, 7412.35samples/s]
[INFO 2017-02-28 22:29:12,117 kur.model.executor:175] Validation loss: 1.641
[INFO 2017-02-28 22:29:12,118 kur.model.executor:422] Saving best historical validation weights: t3/cifar.best.valid.w

Epoch 3/20, loss=1.541: 100%|██████| 10000/10000 [00:02<00:00, 4452.24samples/s]
[INFO 2017-02-28 22:29:14,370 kur.model.executor:390] Training loss: 1.541
[INFO 2017-02-28 22:29:14,371 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.548: 100%|████████| 1600/1600 [00:00<00:00, 8700.24samples/s]
[INFO 2017-02-28 22:29:14,562 kur.model.executor:175] Validation loss: 1.548
[INFO 2017-02-28 22:29:14,562 kur.model.executor:422] Saving best historical validation weights: t3/cifar.best.valid.w

Epoch 4/20, loss=1.444: 100%|██████| 10000/10000 [00:01<00:00, 5273.62samples/s]
[INFO 2017-02-28 22:29:16,462 kur.model.executor:390] Training loss: 1.444
[INFO 2017-02-28 22:29:16,462 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.510: 100%|████████| 1600/1600 [00:00<00:00, 8556.89samples/s]
[INFO 2017-02-28 22:29:16,654 kur.model.executor:175] Validation loss: 1.510
[INFO 2017-02-28 22:29:16,654 kur.model.executor:422] Saving best historical validation weights: t3/cifar.best.valid.w

Epoch 5/20, loss=1.376: 100%|██████| 10000/10000 [00:02<00:00, 4149.76samples/s]
[INFO 2017-02-28 22:29:19,075 kur.model.executor:390] Training loss: 1.376
[INFO 2017-02-28 22:29:19,075 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.431: 100%|████████| 1600/1600 [00:00<00:00, 4356.86samples/s]
[INFO 2017-02-28 22:29:19,448 kur.model.executor:175] Validation loss: 1.431
[INFO 2017-02-28 22:29:19,448 kur.model.executor:422] Saving best historical validation weights: t3/cifar.best.valid.w

Epoch 6/20, loss=1.319: 100%|██████| 10000/10000 [00:02<00:00, 3406.46samples/s]
[INFO 2017-02-28 22:29:22,388 kur.model.executor:390] Training loss: 1.319
[INFO 2017-02-28 22:29:22,388 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.411: 100%|████████| 1600/1600 [00:00<00:00, 4413.76samples/s]
[INFO 2017-02-28 22:29:22,758 kur.model.executor:175] Validation loss: 1.411
[INFO 2017-02-28 22:29:22,758 kur.model.executor:422] Saving best historical validation weights: t3/cifar.best.valid.w

Epoch 7/20, loss=1.265: 100%|██████| 10000/10000 [00:02<00:00, 4796.83samples/s]
[INFO 2017-02-28 22:29:24,846 kur.model.executor:390] Training loss: 1.265
[INFO 2017-02-28 22:29:24,847 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.365: 100%|████████| 1600/1600 [00:00<00:00, 5337.14samples/s]
[INFO 2017-02-28 22:29:25,151 kur.model.executor:175] Validation loss: 1.365
[INFO 2017-02-28 22:29:25,151 kur.model.executor:422] Saving best historical validation weights: t3/cifar.best.valid.w

Epoch 8/20, loss=1.226: 100%|██████| 10000/10000 [00:02<00:00, 4854.19samples/s]
[INFO 2017-02-28 22:29:27,215 kur.model.executor:390] Training loss: 1.226
[INFO 2017-02-28 22:29:27,215 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.401: 100%|████████| 1600/1600 [00:00<00:00, 6717.13samples/s]
[INFO 2017-02-28 22:29:27,459 kur.model.executor:175] Validation loss: 1.401

Epoch 9/20, loss=1.189: 100%|██████| 10000/10000 [00:02<00:00, 4833.21samples/s]
[INFO 2017-02-28 22:29:29,530 kur.model.executor:390] Training loss: 1.189
[INFO 2017-02-28 22:29:29,530 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.325: 100%|████████| 1600/1600 [00:00<00:00, 6882.16samples/s]
[INFO 2017-02-28 22:29:29,768 kur.model.executor:175] Validation loss: 1.325
[INFO 2017-02-28 22:29:29,769 kur.model.executor:422] Saving best historical validation weights: t3/cifar.best.valid.w

Epoch 10/20, loss=1.155: 100%|█████| 10000/10000 [00:02<00:00, 4126.60samples/s]
[INFO 2017-02-28 22:29:32,198 kur.model.executor:390] Training loss: 1.155
[INFO 2017-02-28 22:29:32,198 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.275: 100%|████████| 1600/1600 [00:00<00:00, 6558.35samples/s]
[INFO 2017-02-28 22:29:32,449 kur.model.executor:175] Validation loss: 1.275
[INFO 2017-02-28 22:29:32,449 kur.model.executor:422] Saving best historical validation weights: t3/cifar.best.valid.w

Epoch 11/20, loss=1.124: 100%|█████| 10000/10000 [00:02<00:00, 4612.79samples/s]
[INFO 2017-02-28 22:29:34,624 kur.model.executor:390] Training loss: 1.124
[INFO 2017-02-28 22:29:34,624 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.297: 100%|████████| 1600/1600 [00:00<00:00, 7161.01samples/s]
[INFO 2017-02-28 22:29:34,852 kur.model.executor:175] Validation loss: 1.297

Epoch 12/20, loss=1.093: 100%|█████| 10000/10000 [00:02<00:00, 4749.13samples/s]
[INFO 2017-02-28 22:29:36,960 kur.model.executor:390] Training loss: 1.093
[INFO 2017-02-28 22:29:36,961 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.298: 100%|████████| 1600/1600 [00:00<00:00, 6370.80samples/s]
[INFO 2017-02-28 22:29:37,220 kur.model.executor:175] Validation loss: 1.298

Epoch 13/20, loss=1.064: 100%|█████| 10000/10000 [00:02<00:00, 3999.67samples/s]
[INFO 2017-02-28 22:29:39,723 kur.model.executor:390] Training loss: 1.064
[INFO 2017-02-28 22:29:39,723 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.282: 100%|████████| 1600/1600 [00:00<00:00, 6927.25samples/s]
[INFO 2017-02-28 22:29:39,959 kur.model.executor:175] Validation loss: 1.282

Epoch 14/20, loss=1.049: 100%|█████| 10000/10000 [00:02<00:00, 4225.23samples/s]
[INFO 2017-02-28 22:29:42,327 kur.model.executor:390] Training loss: 1.049
[INFO 2017-02-28 22:29:42,328 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.285: 100%|████████| 1600/1600 [00:00<00:00, 4020.90samples/s]
[INFO 2017-02-28 22:29:42,737 kur.model.executor:175] Validation loss: 1.285

Epoch 15/20, loss=1.029: 100%|█████| 10000/10000 [00:03<00:00, 3258.72samples/s]
[INFO 2017-02-28 22:29:45,808 kur.model.executor:390] Training loss: 1.029
[INFO 2017-02-28 22:29:45,808 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.313: 100%|████████| 1600/1600 [00:00<00:00, 3002.92samples/s]
[INFO 2017-02-28 22:29:46,346 kur.model.executor:175] Validation loss: 1.313

Epoch 16/20, loss=1.000: 100%|█████| 10000/10000 [00:02<00:00, 3371.96samples/s]
[INFO 2017-02-28 22:29:49,314 kur.model.executor:390] Training loss: 1.000
[INFO 2017-02-28 22:29:49,314 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.244: 100%|████████| 1600/1600 [00:00<00:00, 4174.28samples/s]
[INFO 2017-02-28 22:29:49,705 kur.model.executor:175] Validation loss: 1.244
[INFO 2017-02-28 22:29:49,706 kur.model.executor:422] Saving best historical validation weights: t3/cifar.best.valid.w

Epoch 17/20, loss=0.986: 100%|█████| 10000/10000 [00:02<00:00, 4469.39samples/s]
[INFO 2017-02-28 22:29:51,949 kur.model.executor:390] Training loss: 0.986
[INFO 2017-02-28 22:29:51,949 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.294: 100%|████████| 1600/1600 [00:00<00:00, 8014.51samples/s]
[INFO 2017-02-28 22:29:52,154 kur.model.executor:175] Validation loss: 1.294

Epoch 18/20, loss=0.968: 100%|█████| 10000/10000 [00:02<00:00, 4853.69samples/s]
[INFO 2017-02-28 22:29:54,217 kur.model.executor:390] Training loss: 0.968
[INFO 2017-02-28 22:29:54,217 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.352: 100%|████████| 1600/1600 [00:00<00:00, 7541.86samples/s]
[INFO 2017-02-28 22:29:54,435 kur.model.executor:175] Validation loss: 1.352

Epoch 19/20, loss=0.963: 100%|█████| 10000/10000 [00:02<00:00, 4310.08samples/s]
[INFO 2017-02-28 22:29:56,757 kur.model.executor:390] Training loss: 0.963
[INFO 2017-02-28 22:29:56,757 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.215: 100%|████████| 1600/1600 [00:00<00:00, 5476.13samples/s]
[INFO 2017-02-28 22:29:57,060 kur.model.executor:175] Validation loss: 1.215
[INFO 2017-02-28 22:29:57,060 kur.model.executor:422] Saving best historical validation weights: t3/cifar.best.valid.w

Epoch 20/20, loss=0.930: 100%|█████| 10000/10000 [00:02<00:00, 4897.10samples/s]
[INFO 2017-02-28 22:29:59,109 kur.model.executor:390] Training loss: 0.930
[INFO 2017-02-28 22:29:59,109 kur.model.executor:397] Saving best historical training weights: t3/cifar.best.train.w
Validating, loss=1.259: 100%|████████| 1600/1600 [00:00<00:00, 8264.63samples/s]
[INFO 2017-02-28 22:29:59,308 kur.model.executor:175] Validation loss: 1.259
Completed 20 epochs.
[INFO 2017-02-28 22:29:59,309 kur.model.executor:210] Saving most recent weights: t3/cifar.last.w
In [22]:
# %cd t3/
!ls cifar-log
training_loss_labels   validation_loss_labels
training_loss_total    validation_loss_total
In [23]:
from kur.loggers import BinaryLogger
training_loss = BinaryLogger.load_column('cifar-log', 'training_loss_total') 
validation_loss = BinaryLogger.load_column('cifar-log', 'validation_loss_total') 
 
training_loss
Out[23]:
array([ 2.0347259 ,  1.6929394 ,  1.54147506,  1.44420362,  1.3758949 ,
        1.31863678,  1.26508343,  1.2262063 ,  1.18933189,  1.15463436,
        1.12429321,  1.09335482,  1.06431651,  1.04932415,  1.02948546,
        0.99968618,  0.98606402,  0.9677121 ,  0.96266526,  0.93041307], dtype=float32)
In [24]:
import matplotlib.pyplot as plt
plt.xlabel('Epoch')
plt.ylabel('Loss')
epoch = list(range(1, 1+len(training_loss)))
t_line, = plt.plot(epoch, training_loss, 'co-', label='Training Loss')
v_line, = plt.plot(epoch, validation_loss, 'mo-', label='Validation Loss')
plt.legend(handles=[t_line, v_line])
plt.show()
In [4]:
%%writefile dlnd_p2_dropout.yml

---
settings:

  # Where to get the data
  cifar: &cifar
    url: "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
    checksum: "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce"
    path: "~/kur"           # if kur does not have normalization or one-hot-encoding, ##################
                            # without normalization and one-hot-encoding, the performance will be hurt, right? ####
  # Backend to use                                      
  backend:
    name: keras

  # Hyperparameters
  cnn:
    kernels: 20
    size: [5, 5]
    strides: [2, 2]                

  pool:
    size: [2,2]
    strides: [2,2]
    type: max                         # must use a string here, {max} won't work, doc didn't say it #############

model:
  - input: images                       # images are normalized from 0 to 1, labels are one-hot-encoding ##########
  - convolution:                        # does kur do normalize and one-hot-encoding under the hood? ##############
      kernels: "{{cnn.kernels}}"
      size: "{{cnn.size}}"
      strides: "{{ cnn.strides }}"
  - activation: relu
  - pool:
      size: "{{pool.size}}"
      strides: "{{pool.strides}}"
      type: "{{pool.type}}"             
  - flatten:
  - dense: 15                           # p2 want a dropout applied here, but kur does not have yet?? ############
  - dropout: 0.25
  - dense: 10
  - activation: softmax
    name: labels

train:
  data:
    - cifar:
        <<: *cifar
        parts: [1]                      # only use dataset part 1 to train
  provider:
    batch_size: 128
    num_batches: 1000                   # the entire part 1 will be used
  log: t1_dp0.25/cifar-log                   
  epochs: 20
  weights:
    initial: t1_dp0.25/cifar.best.valid.w    
    best: t1_dp0.25/cifar.best.train.w
    last: t1_dp0.25/cifar.last.w

  optimizer:
    name: adam
    learning_rate: 0.001

validate:
  data:
    - cifar:
       <<: *cifar
       parts: 5                          # only use dataset part 5 as validation set
  provider:
    batch_size: 128
    num_batches: 50                      # the project 2 only used 5000 data points as validation set
  weights: t1_dp0.25/cifar.best.valid.w

test: &test
  data:
    - cifar:
       <<: *cifar
       parts: test
  weights: t1_dp0.25/cifar.best.valid.w
  provider:
    batch_size: 128
    num_batches: 1000                     # the entire part test will be used

evaluate:
  <<: *test
  destination: t1_dp0.25/cifar.results.pkl

loss:
  - target: labels                        # in the project: training loss and valid_accuracy are printed #############
    name: categorical_crossentropy        # this should be a matter of personal taste, won't really affect anything##
...
Overwriting dlnd_p2_dropout.yml
In [5]:
!kur -v train dlnd_p2_dropout.yml
[INFO 2017-03-01 07:34:45,517 kur.kurfile:699] Parsing source: dlnd_p2_dropout.yml, included by top-level.
[INFO 2017-03-01 07:34:45,533 kur.kurfile:82] Parsing Kurfile...
[INFO 2017-03-01 07:34:45,558 kur.loggers.binary_logger:107] Log does not exist. Creating path: t1_dp0.25/cifar-log
[INFO 2017-03-01 07:34:58,394 kur.backend.backend:80] Creating backend: keras
[INFO 2017-03-01 07:34:58,394 kur.backend.backend:83] Backend variants: none
[INFO 2017-03-01 07:34:58,394 kur.backend.keras_backend:122] No particular backend for Keras has been requested.
[INFO 2017-03-01 07:34:59,341 kur.backend.keras_backend:195] Keras is loaded. The backend is: theano
[INFO 2017-03-01 07:34:59,341 kur.model.model:260] Enumerating the model containers.
[INFO 2017-03-01 07:34:59,342 kur.model.model:265] Assembling the model dependency graph.
[INFO 2017-03-01 07:34:59,342 kur.model.model:280] Connecting the model graph.
[INFO 2017-03-01 07:35:00,404 kur.model.model:284] Model inputs:  images
[INFO 2017-03-01 07:35:00,404 kur.model.model:285] Model outputs: labels
[INFO 2017-03-01 07:35:00,404 kur.kurfile:357] Ignoring missing initial weights: t1_dp0.25/cifar.best.valid.w. If this is undesireable, set "must_exist" to "yes" in the approriate "weights" section.
[INFO 2017-03-01 07:35:00,404 kur.model.executor:315] No historical training loss available from logs.
[INFO 2017-03-01 07:35:00,404 kur.model.executor:323] No historical validation loss available from logs.
[INFO 2017-03-01 07:35:00,404 kur.model.executor:329] No previous epochs.
[INFO 2017-03-01 07:35:01,829 kur.backend.keras_backend:666] Waiting for model to finish compiling...

Epoch 1/20, loss=2.080: 100%|██████| 10000/10000 [00:02<00:00, 3728.18samples/s]
[INFO 2017-03-01 07:35:04,562 kur.model.executor:464] Training loss: 2.080
[INFO 2017-03-01 07:35:04,562 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
[INFO 2017-03-01 07:35:04,953 kur.backend.keras_backend:666] Waiting for model to finish compiling...
Validating, loss=1.874: 100%|███████| 6400/6400 [00:00<00:00, 10436.54samples/s]
[INFO 2017-03-01 07:35:05,569 kur.model.executor:197] Validation loss: 1.874
[INFO 2017-03-01 07:35:05,570 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 2/20, loss=1.801: 100%|██████| 10000/10000 [00:02<00:00, 4985.77samples/s]
[INFO 2017-03-01 07:35:07,580 kur.model.executor:464] Training loss: 1.801
[INFO 2017-03-01 07:35:07,580 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.672: 100%|███████| 6400/6400 [00:00<00:00, 10588.70samples/s]
[INFO 2017-03-01 07:35:08,196 kur.model.executor:197] Validation loss: 1.672
[INFO 2017-03-01 07:35:08,196 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 3/20, loss=1.646: 100%|██████| 10000/10000 [00:02<00:00, 4988.95samples/s]
[INFO 2017-03-01 07:35:10,206 kur.model.executor:464] Training loss: 1.646
[INFO 2017-03-01 07:35:10,206 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.578: 100%|███████| 6400/6400 [00:00<00:00, 11537.31samples/s]
[INFO 2017-03-01 07:35:10,768 kur.model.executor:197] Validation loss: 1.578
[INFO 2017-03-01 07:35:10,768 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 4/20, loss=1.537: 100%|██████| 10000/10000 [00:02<00:00, 4735.37samples/s]
[INFO 2017-03-01 07:35:12,886 kur.model.executor:464] Training loss: 1.537
[INFO 2017-03-01 07:35:12,887 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.490: 100%|███████| 6400/6400 [00:00<00:00, 10859.18samples/s]
[INFO 2017-03-01 07:35:13,486 kur.model.executor:197] Validation loss: 1.490
[INFO 2017-03-01 07:35:13,486 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 5/20, loss=1.470: 100%|██████| 10000/10000 [00:02<00:00, 4768.44samples/s]
[INFO 2017-03-01 07:35:15,588 kur.model.executor:464] Training loss: 1.470
[INFO 2017-03-01 07:35:15,588 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.411: 100%|███████| 6400/6400 [00:00<00:00, 10774.77samples/s]
[INFO 2017-03-01 07:35:16,189 kur.model.executor:197] Validation loss: 1.411
[INFO 2017-03-01 07:35:16,190 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 6/20, loss=1.406: 100%|██████| 10000/10000 [00:02<00:00, 4293.21samples/s]
[INFO 2017-03-01 07:35:18,523 kur.model.executor:464] Training loss: 1.406
[INFO 2017-03-01 07:35:18,524 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.384: 100%|███████| 6400/6400 [00:00<00:00, 10308.95samples/s]
[INFO 2017-03-01 07:35:19,153 kur.model.executor:197] Validation loss: 1.384
[INFO 2017-03-01 07:35:19,154 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 7/20, loss=1.371: 100%|██████| 10000/10000 [00:02<00:00, 4549.75samples/s]
[INFO 2017-03-01 07:35:21,358 kur.model.executor:464] Training loss: 1.371
[INFO 2017-03-01 07:35:21,358 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.368: 100%|████████| 6400/6400 [00:00<00:00, 4923.30samples/s]
[INFO 2017-03-01 07:35:21,993 kur.model.executor:197] Validation loss: 1.368
[INFO 2017-03-01 07:35:21,994 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 8/20, loss=1.330: 100%|██████| 10000/10000 [00:02<00:00, 4738.82samples/s]
[INFO 2017-03-01 07:35:24,111 kur.model.executor:464] Training loss: 1.330
[INFO 2017-03-01 07:35:24,111 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.328: 100%|███████| 6400/6400 [00:00<00:00, 10351.03samples/s]
[INFO 2017-03-01 07:35:24,737 kur.model.executor:197] Validation loss: 1.328
[INFO 2017-03-01 07:35:24,738 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 9/20, loss=1.303: 100%|██████| 10000/10000 [00:02<00:00, 4794.48samples/s]
[INFO 2017-03-01 07:35:26,830 kur.model.executor:464] Training loss: 1.303
[INFO 2017-03-01 07:35:26,830 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.339: 100%|███████| 6400/6400 [00:00<00:00, 11131.73samples/s]
[INFO 2017-03-01 07:35:27,412 kur.model.executor:197] Validation loss: 1.339

Epoch 10/20, loss=1.294: 100%|█████| 10000/10000 [00:02<00:00, 4693.60samples/s]
[INFO 2017-03-01 07:35:29,546 kur.model.executor:464] Training loss: 1.294
[INFO 2017-03-01 07:35:29,546 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.310: 100%|███████| 6400/6400 [00:00<00:00, 11321.00samples/s]
[INFO 2017-03-01 07:35:30,118 kur.model.executor:197] Validation loss: 1.310
[INFO 2017-03-01 07:35:30,119 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 11/20, loss=1.253: 100%|█████| 10000/10000 [00:02<00:00, 4719.59samples/s]
[INFO 2017-03-01 07:35:32,243 kur.model.executor:464] Training loss: 1.253
[INFO 2017-03-01 07:35:32,243 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.293: 100%|███████| 6400/6400 [00:00<00:00, 10417.43samples/s]
[INFO 2017-03-01 07:35:32,866 kur.model.executor:197] Validation loss: 1.293
[INFO 2017-03-01 07:35:32,867 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 12/20, loss=1.233: 100%|█████| 10000/10000 [00:02<00:00, 4647.94samples/s]
[INFO 2017-03-01 07:35:35,024 kur.model.executor:464] Training loss: 1.233
[INFO 2017-03-01 07:35:35,024 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.268: 100%|████████| 6400/6400 [00:00<00:00, 9125.34samples/s]
[INFO 2017-03-01 07:35:35,748 kur.model.executor:197] Validation loss: 1.268
[INFO 2017-03-01 07:35:35,748 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 13/20, loss=1.213: 100%|█████| 10000/10000 [00:02<00:00, 4052.58samples/s]
[INFO 2017-03-01 07:35:38,221 kur.model.executor:464] Training loss: 1.213
[INFO 2017-03-01 07:35:38,221 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.263: 100%|███████| 6400/6400 [00:00<00:00, 10334.80samples/s]
[INFO 2017-03-01 07:35:38,848 kur.model.executor:197] Validation loss: 1.263
[INFO 2017-03-01 07:35:38,848 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 14/20, loss=1.192: 100%|█████| 10000/10000 [00:02<00:00, 4059.79samples/s]
[INFO 2017-03-01 07:35:41,316 kur.model.executor:464] Training loss: 1.192
[INFO 2017-03-01 07:35:41,316 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.267: 100%|███████| 6400/6400 [00:00<00:00, 10624.42samples/s]
[INFO 2017-03-01 07:35:41,927 kur.model.executor:197] Validation loss: 1.267

Epoch 15/20, loss=1.179: 100%|█████| 10000/10000 [00:03<00:00, 3174.28samples/s]
[INFO 2017-03-01 07:35:45,081 kur.model.executor:464] Training loss: 1.179
[INFO 2017-03-01 07:35:45,081 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.250: 100%|███████| 6400/6400 [00:00<00:00, 10746.89samples/s]
[INFO 2017-03-01 07:35:45,686 kur.model.executor:197] Validation loss: 1.250
[INFO 2017-03-01 07:35:45,686 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 16/20, loss=1.155: 100%|█████| 10000/10000 [00:02<00:00, 3838.08samples/s]
[INFO 2017-03-01 07:35:48,298 kur.model.executor:464] Training loss: 1.155
[INFO 2017-03-01 07:35:48,298 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.249: 100%|███████| 6400/6400 [00:00<00:00, 10454.68samples/s]
[INFO 2017-03-01 07:35:48,938 kur.model.executor:197] Validation loss: 1.249
[INFO 2017-03-01 07:35:48,939 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 17/20, loss=1.141: 100%|█████| 10000/10000 [00:02<00:00, 3516.54samples/s]
[INFO 2017-03-01 07:35:51,791 kur.model.executor:464] Training loss: 1.141
[INFO 2017-03-01 07:35:51,791 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.235: 100%|███████| 6400/6400 [00:00<00:00, 10439.16samples/s]
[INFO 2017-03-01 07:35:52,412 kur.model.executor:197] Validation loss: 1.235
[INFO 2017-03-01 07:35:52,412 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 18/20, loss=1.123: 100%|█████| 10000/10000 [00:03<00:00, 3276.45samples/s]
[INFO 2017-03-01 07:35:55,469 kur.model.executor:464] Training loss: 1.123
[INFO 2017-03-01 07:35:55,470 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.238: 100%|████████| 6400/6400 [00:00<00:00, 7017.06samples/s]
[INFO 2017-03-01 07:35:56,390 kur.model.executor:197] Validation loss: 1.238

Epoch 19/20, loss=1.121: 100%|█████| 10000/10000 [00:02<00:00, 4330.83samples/s]
[INFO 2017-03-01 07:35:58,703 kur.model.executor:464] Training loss: 1.121
[INFO 2017-03-01 07:35:58,704 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.224: 100%|███████| 6400/6400 [00:00<00:00, 11403.63samples/s]
[INFO 2017-03-01 07:35:59,272 kur.model.executor:197] Validation loss: 1.224
[INFO 2017-03-01 07:35:59,272 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 20/20, loss=1.107: 100%|█████| 10000/10000 [00:02<00:00, 4730.36samples/s]
[INFO 2017-03-01 07:36:01,393 kur.model.executor:464] Training loss: 1.107
[INFO 2017-03-01 07:36:01,394 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.234: 100%|███████| 6400/6400 [00:00<00:00, 11209.60samples/s]
[INFO 2017-03-01 07:36:01,972 kur.model.executor:197] Validation loss: 1.234
Completed 20 epochs.
[INFO 2017-03-01 07:36:01,976 kur.model.executor:235] Saving most recent weights: t1_dp0.25/cifar.last.w
In [7]:
# %cd t1_dp0.25/
!ls cifar-log
batch_loss_batch       training_loss_batch    validation_loss_labels
batch_loss_labels      training_loss_labels   validation_loss_total
batch_loss_total       training_loss_total
summary.yml            validation_loss_batch
In [9]:
%pycat summary.yml
ERROR:root:Cell magic `%%pycat` not found (But line magic `%pycat` exists, did you mean that instead?).
In [10]:
from kur.loggers import BinaryLogger
training_loss = BinaryLogger.load_column('cifar-log', 'training_loss_total') 
validation_loss = BinaryLogger.load_column('cifar-log', 'validation_loss_total') 
 
training_loss
Out[10]:
array([ 2.08002758,  1.80103576,  1.64649773,  1.53694201,  1.47005808,
        1.4060986 ,  1.37083662,  1.32995439,  1.3027122 ,  1.29357958,
        1.25282907,  1.23259354,  1.21278262,  1.19201612,  1.17865086,
        1.15514135,  1.14136755,  1.1234889 ,  1.1214298 ,  1.10675633], dtype=float32)
In [11]:
import matplotlib.pyplot as plt
plt.xlabel('Epoch')
plt.ylabel('Loss')
epoch = list(range(1, 1+len(training_loss)))
t_line, = plt.plot(epoch, training_loss, 'co-', label='Training Loss')
v_line, = plt.plot(epoch, validation_loss, 'mo-', label='Validation Loss')
plt.legend(handles=[t_line, v_line])
plt.show()
In [ ]:
 

Learning Tasks on Kur

Task1: Split kurfile

  • split kurfile into fluid and default parts
  • default part remain constant
  • fluid part will be easy to change constantly for experimenting

Solution1: Split kurfile

  • layer parameters will keep changing
  • file path will keep changing
  • data parts for train and validate will change
  • batch_size and num_batches will change
  • hyperparameters like learning rate will change

Create a dlnd_p2_defaults.yml

In [4]:
%%writefile dlnd_p2_defaults.yml

---
settings:

  # Where to get the data
  cifar: &cifar
    url: "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
    checksum: "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce"
    path: "~/kur"  
    
  # Backend to use                                      
  backend:
    name: keras    
    
model:                             # there should be loop design for model structure flexibility
  - input: images                       
  - convolution:                        
      kernels: "{{cnn.kernels}}"
      size: "{{cnn.size}}"
      strides: "{{ cnn.strides }}"
  - activation: relu
  - pool:
      size: "{{pool.size}}"
      strides: "{{pool.strides}}"
      type: "{{pool.type}}"             
  - flatten:
  - dense: "{{dense.out1}}"       # 15                    
  - dense: "{{dense.out2}}"       # 10
  - activation: softmax
    name: labels    
    

    
train:
  data:
    - cifar:
        <<: *cifar
        parts: "{{part.train}}"               # only use dataset part 1 to train
  provider:
    batch_size: "{{batch_size.train}}"        # 128
    num_batches: "{{num_batches.train}}"      # total batch is less than 100, so use 1000 is to select all       #####     
    randomize: True                           # Importance: when train on small sample we still want weights to have 
                                              # generalize power. model overfit by memorizing the same data points#### 

  log: "{{path.log}}"                         # t1_dp0.25/cifar-log                   
  epochs: "{{num_epochs}}"                    # 20
  weights:
    initial: "{{path.initial_w}}"             # t1_dp0.25/cifar.best.valid.w    
    best: "{{path.best_train_w}}"             # t1_dp0.25/cifar.best.train.w
    last: "{{path.last_w}}"                   # t1_dp0.25/cifar.last.w

  optimizer:
    name: adam
    learning_rate: "{{learning_rate}}"        # 0.001       


validate:
  data:
    - cifar:
       <<: *cifar
       parts: "{{part.valid}}"                # only use dataset part 5 as validation set
  provider:
    batch_size: "{{batch_size.valid}}"        # same as training, 128
    num_batches: "{{num_batches.valid}}"      # the project 2 only used 5000 data points as validation set, so 50 
  weights: "{{path.best_valid_w}}"            # t1_dp0.25/cifar.best.valid.w

test: &test
  data:
    - cifar:
       <<: *cifar
       parts: test
  weights: "{{path.best_valid_w}}"            # t1_dp0.25/cifar.best.valid.w
  provider:
    batch_size: "{{batch_size.test}}"         # same as training, 128
    num_batches: "{{num_batches.test}}"      # the entire part test will be used, set it 1000

evaluate:
  <<: *test
  destination: "{{path.result}}"              # t1_dp0.25/cifar.results.pkl

loss:
  - target: labels                       
    name: categorical_crossentropy        

...
Overwriting dlnd_p2_defaults.yml

Create dlnd_p2_fluid.yml

In [5]:
%%writefile dlnd_p2_fluid.yml

---
settings: 
  cnn: 
    kernels: 20              # why cannot use [20]? what if there are 2 conv layers [20, 30]
    size: [5,5]  
    strides: [2,2]
  pool: 
    size: [2,2]
    strides: [2,2]
    type: "max"  
  dense: 
    out1: 15
    out2: 10
  part: 
    train: 1
    valid: 5
    test: test
  batch_size: 
    train: 128
    valid: 128
    test: 128
  num_batches: 
    train:                # 1000 and nothing here both refer to all the batches? ###########################
    valid: 50
    test: 
  path: 
    log: t1_dp0.25/cifar-log
    initial_w: t1_dp0.25/cifar.best.valid.w
    best_train_w: t1_dp0.25/cifar.best.train.w
    last_w: t1_dp0.25/cifar.last.w
    best_valid_w: t1_dp0.25/cifar.best.valid.w
    result: t1_dp0.25/cifar.results.pkl
  learning_rate: 0.001
  num_epochs: 1


include: dlnd_p2_defaults.yml  # to define default setting
...
Overwriting dlnd_p2_fluid.yml
  • is grid search or random search for best parameters or hyperparameter is a feature kur want to implement?
  • automatically searching for best parameters like gradient descent search for local or gloabl optimal is a nice thing to have right?
In [6]:
!kur -vv train dlnd_p2_fluid.yml
[INFO 2017-03-01 15:24:19,985 kur.kurfile:699] Parsing source: dlnd_p2_fluid.yml, included by top-level.
[INFO 2017-03-01 15:24:19,996 kur.kurfile:699] Parsing source: dlnd_p2_defaults.yml, included by dlnd_p2_fluid.yml.
[INFO 2017-03-01 15:24:20,010 kur.kurfile:82] Parsing Kurfile...
[DEBUG 2017-03-01 15:24:20,010 kur.kurfile:784] Parsing Kurfile section: settings
[DEBUG 2017-03-01 15:24:20,016 kur.kurfile:784] Parsing Kurfile section: train
[DEBUG 2017-03-01 15:24:20,023 kur.kurfile:784] Parsing Kurfile section: validate
[DEBUG 2017-03-01 15:24:20,028 kur.kurfile:784] Parsing Kurfile section: test
[DEBUG 2017-03-01 15:24:20,033 kur.kurfile:784] Parsing Kurfile section: evaluate
[DEBUG 2017-03-01 15:24:20,039 kur.containers.layers.placeholder:63] Using short-hand name for placeholder: images
[DEBUG 2017-03-01 15:24:20,040 kur.containers.layers.placeholder:97] Placeholder "images" has a deferred shape.
[DEBUG 2017-03-01 15:24:20,050 kur.kurfile:784] Parsing Kurfile section: loss
[INFO 2017-03-01 15:24:20,052 kur.loggers.binary_logger:107] Log does not exist. Creating path: t1_dp0.25/cifar-log
[DEBUG 2017-03-01 15:24:20,980 kur.utils.package:233] File exists and passed checksum: /Users/Natsume/kur/cifar-10-python.tar.gz
[DEBUG 2017-03-01 15:24:27,155 kur.providers.batch_provider:57] Batch size set to: 128
[DEBUG 2017-03-01 15:24:27,863 kur.utils.package:233] File exists and passed checksum: /Users/Natsume/kur/cifar-10-python.tar.gz
[DEBUG 2017-03-01 15:24:33,548 kur.providers.batch_provider:57] Batch size set to: 128
[DEBUG 2017-03-01 15:24:33,548 kur.providers.batch_provider:102] Maximum number of batches set to: 50
[DEBUG 2017-03-01 15:24:33,548 kur.backend.backend:187] Using backend: keras
[INFO 2017-03-01 15:24:33,549 kur.backend.backend:80] Creating backend: keras
[INFO 2017-03-01 15:24:33,549 kur.backend.backend:83] Backend variants: none
[INFO 2017-03-01 15:24:33,549 kur.backend.keras_backend:122] No particular backend for Keras has been requested.
[DEBUG 2017-03-01 15:24:33,549 kur.backend.keras_backend:124] Using the system-default Keras backend.
[DEBUG 2017-03-01 15:24:33,549 kur.backend.keras_backend:189] Overriding environmental variables: {'TF_CPP_MIN_LOG_LEVEL': '1', 'KERAS_BACKEND': None, 'THEANO_FLAGS': None}
[INFO 2017-03-01 15:24:34,503 kur.backend.keras_backend:195] Keras is loaded. The backend is: theano
[INFO 2017-03-01 15:24:34,503 kur.model.model:260] Enumerating the model containers.
[INFO 2017-03-01 15:24:34,503 kur.model.model:265] Assembling the model dependency graph.
[DEBUG 2017-03-01 15:24:34,503 kur.model.model:272] Assembled Node: images
[DEBUG 2017-03-01 15:24:34,503 kur.model.model:274]   Uses: 
[DEBUG 2017-03-01 15:24:34,503 kur.model.model:276]   Used by: ..convolution.0
[DEBUG 2017-03-01 15:24:34,503 kur.model.model:277]   Aliases: images
[DEBUG 2017-03-01 15:24:34,503 kur.model.model:272] Assembled Node: ..convolution.0
[DEBUG 2017-03-01 15:24:34,503 kur.model.model:274]   Uses: images
[DEBUG 2017-03-01 15:24:34,503 kur.model.model:276]   Used by: ..activation.0
[DEBUG 2017-03-01 15:24:34,503 kur.model.model:277]   Aliases: ..convolution.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:272] Assembled Node: ..activation.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:274]   Uses: ..convolution.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:276]   Used by: ..pool.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:277]   Aliases: ..activation.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:272] Assembled Node: ..pool.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:274]   Uses: ..activation.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:276]   Used by: ..flatten.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:277]   Aliases: ..pool.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:272] Assembled Node: ..flatten.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:274]   Uses: ..pool.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:276]   Used by: ..dense.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:277]   Aliases: ..flatten.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:272] Assembled Node: ..dense.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:274]   Uses: ..flatten.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:276]   Used by: ..dense.1
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:277]   Aliases: ..dense.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:272] Assembled Node: ..dense.1
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:274]   Uses: ..dense.0
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:276]   Used by: labels
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:277]   Aliases: ..dense.1
[DEBUG 2017-03-01 15:24:34,504 kur.model.model:272] Assembled Node: labels
[DEBUG 2017-03-01 15:24:34,505 kur.model.model:274]   Uses: ..dense.1
[DEBUG 2017-03-01 15:24:34,505 kur.model.model:276]   Used by: 
[DEBUG 2017-03-01 15:24:34,505 kur.model.model:277]   Aliases: labels
[INFO 2017-03-01 15:24:34,505 kur.model.model:280] Connecting the model graph.
[DEBUG 2017-03-01 15:24:34,505 kur.model.model:311] Building node: images
[DEBUG 2017-03-01 15:24:34,505 kur.model.model:312]   Aliases: images
[DEBUG 2017-03-01 15:24:34,505 kur.model.model:313]   Inputs:
[DEBUG 2017-03-01 15:24:34,505 kur.containers.layers.placeholder:117] Creating placeholder for "images" with data type "float32".
[DEBUG 2017-03-01 15:24:34,505 kur.model.model:125] Trying to infer shape for input "images"
[DEBUG 2017-03-01 15:24:34,505 kur.model.model:143] Inferred shape for input "images": (32, 32, 3)
[DEBUG 2017-03-01 15:24:34,505 kur.containers.layers.placeholder:127] Inferred shape: (32, 32, 3)
[DEBUG 2017-03-01 15:24:34,510 kur.model.model:382]   Value: images
[DEBUG 2017-03-01 15:24:34,510 kur.model.model:311] Building node: ..convolution.0
[DEBUG 2017-03-01 15:24:34,510 kur.model.model:312]   Aliases: ..convolution.0
[DEBUG 2017-03-01 15:24:34,510 kur.model.model:313]   Inputs:
[DEBUG 2017-03-01 15:24:34,510 kur.model.model:315]   - images: images
[DEBUG 2017-03-01 15:24:35,577 kur.model.model:382]   Value: Elemwise{add,no_inplace}.0
[DEBUG 2017-03-01 15:24:35,577 kur.model.model:311] Building node: ..activation.0
[DEBUG 2017-03-01 15:24:35,577 kur.model.model:312]   Aliases: ..activation.0
[DEBUG 2017-03-01 15:24:35,577 kur.model.model:313]   Inputs:
[DEBUG 2017-03-01 15:24:35,577 kur.model.model:315]   - ..convolution.0: Elemwise{add,no_inplace}.0
[DEBUG 2017-03-01 15:24:35,612 kur.model.model:382]   Value: Elemwise{mul,no_inplace}.0
[DEBUG 2017-03-01 15:24:35,612 kur.model.model:311] Building node: ..pool.0
[DEBUG 2017-03-01 15:24:35,612 kur.model.model:312]   Aliases: ..pool.0
[DEBUG 2017-03-01 15:24:35,612 kur.model.model:313]   Inputs:
[DEBUG 2017-03-01 15:24:35,612 kur.model.model:315]   - ..activation.0: Elemwise{mul,no_inplace}.0
[DEBUG 2017-03-01 15:24:35,615 kur.model.model:382]   Value: DimShuffle{0,2,3,1}.0
[DEBUG 2017-03-01 15:24:35,615 kur.model.model:311] Building node: ..flatten.0
[DEBUG 2017-03-01 15:24:35,615 kur.model.model:312]   Aliases: ..flatten.0
[DEBUG 2017-03-01 15:24:35,615 kur.model.model:313]   Inputs:
[DEBUG 2017-03-01 15:24:35,615 kur.model.model:315]   - ..pool.0: DimShuffle{0,2,3,1}.0
[DEBUG 2017-03-01 15:24:35,622 kur.model.model:382]   Value: Reshape{2}.0
[DEBUG 2017-03-01 15:24:35,622 kur.model.model:311] Building node: ..dense.0
[DEBUG 2017-03-01 15:24:35,622 kur.model.model:312]   Aliases: ..dense.0
[DEBUG 2017-03-01 15:24:35,622 kur.model.model:313]   Inputs:
[DEBUG 2017-03-01 15:24:35,622 kur.model.model:315]   - ..flatten.0: Reshape{2}.0
[DEBUG 2017-03-01 15:24:35,624 kur.model.model:382]   Value: Elemwise{add,no_inplace}.0
[DEBUG 2017-03-01 15:24:35,624 kur.model.model:311] Building node: ..dense.1
[DEBUG 2017-03-01 15:24:35,624 kur.model.model:312]   Aliases: ..dense.1
[DEBUG 2017-03-01 15:24:35,624 kur.model.model:313]   Inputs:
[DEBUG 2017-03-01 15:24:35,624 kur.model.model:315]   - ..dense.0: Elemwise{add,no_inplace}.0
[DEBUG 2017-03-01 15:24:35,625 kur.model.model:382]   Value: Elemwise{add,no_inplace}.0
[DEBUG 2017-03-01 15:24:35,625 kur.model.model:311] Building node: labels
[DEBUG 2017-03-01 15:24:35,625 kur.model.model:312]   Aliases: labels
[DEBUG 2017-03-01 15:24:35,625 kur.model.model:313]   Inputs:
[DEBUG 2017-03-01 15:24:35,625 kur.model.model:315]   - ..dense.1: Elemwise{add,no_inplace}.0
[DEBUG 2017-03-01 15:24:35,625 kur.model.model:382]   Value: Softmax.0
[INFO 2017-03-01 15:24:35,625 kur.model.model:284] Model inputs:  images
[INFO 2017-03-01 15:24:35,625 kur.model.model:285] Model outputs: labels
[INFO 2017-03-01 15:24:35,625 kur.kurfile:357] Ignoring missing initial weights: t1_dp0.25/cifar.best.valid.w. If this is undesireable, set "must_exist" to "yes" in the approriate "weights" section.
[INFO 2017-03-01 15:24:35,625 kur.model.executor:315] No historical training loss available from logs.
[INFO 2017-03-01 15:24:35,626 kur.model.executor:323] No historical validation loss available from logs.
[INFO 2017-03-01 15:24:35,626 kur.model.executor:329] No previous epochs.
[DEBUG 2017-03-01 15:24:35,626 kur.model.executor:353] Epoch handling mode: additional
[DEBUG 2017-03-01 15:24:35,626 kur.model.executor:101] Recompiling the model.
[DEBUG 2017-03-01 15:24:35,626 kur.backend.keras_backend:527] Instantiating a Keras model.
[DEBUG 2017-03-01 15:24:35,825 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-01 15:24:35,825 kur.backend.keras_backend:538] Layer (type)                     Output Shape          Param #     Connected to                     
[DEBUG 2017-03-01 15:24:35,825 kur.backend.keras_backend:538] ====================================================================================================
[DEBUG 2017-03-01 15:24:35,825 kur.backend.keras_backend:538] images (InputLayer)              (None, 32, 32, 3)     0                                            
[DEBUG 2017-03-01 15:24:35,825 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-01 15:24:35,825 kur.backend.keras_backend:538] ..convolution.0 (Convolution2D)  (None, 16, 16, 20)    1520        images[0][0]                     
[DEBUG 2017-03-01 15:24:35,825 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] ..activation.0 (Activation)      (None, 16, 16, 20)    0           ..convolution.0[0][0]            
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] ..pool.0 (MaxPooling2D)          (None, 8, 8, 20)      0           ..activation.0[0][0]             
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] ..flatten.0 (Flatten)            (None, 1280)          0           ..pool.0[0][0]                   
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] ..dense.0 (Dense)                (None, 15)            19215       ..flatten.0[0][0]                
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] ..dense.1 (Dense)                (None, 10)            160         ..dense.0[0][0]                  
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] labels (Activation)              (None, 10)            0           ..dense.1[0][0]                  
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] ====================================================================================================
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] Total params: 20,895
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] Trainable params: 20,895
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] Non-trainable params: 0
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:538] 
[DEBUG 2017-03-01 15:24:35,826 kur.backend.keras_backend:576] Assembling a training function from the model.
[DEBUG 2017-03-01 15:24:35,831 kur.backend.keras_backend:509] Adding additional inputs: labels
[DEBUG 2017-03-01 15:24:37,417 kur.backend.keras_backend:599] Additional inputs for log functions: labels
[DEBUG 2017-03-01 15:24:37,417 kur.backend.keras_backend:616] Expected input shapes: images=(None, 32, 32, 3), labels=(None, None)
[DEBUG 2017-03-01 15:24:37,417 kur.backend.keras_backend:634] Compiled model: {'names': {'output': ['labels', 'labels'], 'input': ['images', 'labels']}, 'func': <keras.backend.theano_backend.Function object at 0x129e70ef0>, 'shapes': {'input': [(None, 32, 32, 3), (None, None)]}}
[DEBUG 2017-03-01 15:24:37,417 kur.providers.batch_provider:57] Batch size set to: 2
[DEBUG 2017-03-01 15:24:37,417 kur.providers.batch_provider:102] Maximum number of batches set to: 1
[INFO 2017-03-01 15:24:37,420 kur.backend.keras_backend:666] Waiting for model to finish compiling...
[DEBUG 2017-03-01 15:24:37,420 kur.providers.batch_provider:139] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:37,420 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:37,469 kur.providers.provider:144] Data source "labels": entries=10000, shape=(10,)
[DEBUG 2017-03-01 15:24:37,470 kur.providers.provider:144] Data source "images": entries=10000, shape=(32, 32, 3)

Epoch 1/1, loss=N/A:   0%|                       | 0/10000 [00:00<?, ?samples/s][DEBUG 2017-03-01 15:24:37,475 kur.providers.shuffle_provider:184] Shuffling...
[DEBUG 2017-03-01 15:24:37,646 kur.providers.batch_provider:139] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:37,647 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:37,647 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:37,656 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:37,681 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:37,682 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:37,682 kur.loggers.binary_logger:135] Adding data to binary column: batch_loss_total
[DEBUG 2017-03-01 15:24:37,683 kur.loggers.binary_logger:135] Adding data to binary column: batch_loss_labels
[DEBUG 2017-03-01 15:24:37,683 kur.loggers.binary_logger:135] Adding data to binary column: batch_loss_batch
[DEBUG 2017-03-01 15:24:37,683 kur.loggers.binary_logger:144] Writing logger summary.
Epoch 1/1, loss=2.321:   1%|▏         | 128/10000 [00:00<00:16, 610.76samples/s][DEBUG 2017-03-01 15:24:37,685 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:37,694 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:37,698 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:37,717 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:37,717 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:37,718 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:37,730 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:37,754 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:37,755 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:37,755 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:37,772 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:37,797 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.325:   5%|▌         | 512/10000 [00:00<00:11, 810.28samples/s][DEBUG 2017-03-01 15:24:37,798 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:37,798 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:37,813 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:37,829 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:37,830 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:37,838 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:37,844 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:37,868 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:37,869 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:37,869 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:37,882 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:37,902 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.314:   9%|▊        | 896/10000 [00:00<00:08, 1057.23samples/s][DEBUG 2017-03-01 15:24:37,902 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:37,903 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:37,916 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:37,938 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:37,940 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:37,951 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:37,957 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:37,977 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:37,977 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:37,977 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:37,990 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,011 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.304:  13%|█       | 1280/10000 [00:00<00:06, 1338.00samples/s][DEBUG 2017-03-01 15:24:38,012 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,012 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,028 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,046 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,047 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,047 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,063 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,086 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,086 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,086 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,097 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,127 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.287:  17%|█▎      | 1664/10000 [00:00<00:05, 1630.08samples/s][DEBUG 2017-03-01 15:24:38,127 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,127 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,137 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,164 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,165 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,165 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,178 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,199 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,199 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,200 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,210 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,234 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.277:  20%|█▋      | 2048/10000 [00:00<00:04, 1947.87samples/s][DEBUG 2017-03-01 15:24:38,235 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,235 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,244 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,270 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,270 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,270 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,278 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,301 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,302 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,302 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,311 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,333 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,334 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,334 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,346 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,370 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.261:  26%|██      | 2560/10000 [00:00<00:03, 2278.82samples/s][DEBUG 2017-03-01 15:24:38,370 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,371 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,384 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,405 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,405 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,405 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,408 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,435 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,436 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,436 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,446 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,472 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.249:  29%|██▎     | 2944/10000 [00:00<00:02, 2584.62samples/s][DEBUG 2017-03-01 15:24:38,472 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,479 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,484 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,500 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,501 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,501 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,512 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,530 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,531 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,539 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,543 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,563 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,564 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,564 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,574 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,591 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.228:  35%|██▊     | 3456/10000 [00:01<00:02, 2937.59samples/s][DEBUG 2017-03-01 15:24:38,591 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,601 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,605 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,626 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,627 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,627 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,638 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,659 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,660 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,660 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,674 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,690 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,691 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,701 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,705 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,723 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.210:  40%|███▏    | 3968/10000 [00:01<00:01, 3167.84samples/s][DEBUG 2017-03-01 15:24:38,723 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,732 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,736 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,754 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,755 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,765 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,770 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,789 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,790 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,790 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,800 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,817 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,818 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,818 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,831 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,848 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.188:  45%|███▌    | 4480/10000 [00:01<00:01, 3398.18samples/s][DEBUG 2017-03-01 15:24:38,848 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,856 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,860 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,887 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,888 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,888 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,903 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,922 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,922 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,923 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,936 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,955 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.178:  49%|███▉    | 4864/10000 [00:01<00:01, 3449.12samples/s][DEBUG 2017-03-01 15:24:38,956 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,956 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:38,970 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:38,990 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:38,990 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:38,998 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,002 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,022 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,023 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,023 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,037 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,057 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.167:  52%|████▏   | 5248/10000 [00:01<00:01, 3540.87samples/s][DEBUG 2017-03-01 15:24:39,057 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,058 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,069 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,089 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,090 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,090 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,101 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,119 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,119 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,130 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,134 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,155 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,156 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,156 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,170 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,192 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.152:  58%|████▌   | 5760/10000 [00:01<00:01, 3609.14samples/s][DEBUG 2017-03-01 15:24:39,193 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,203 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,206 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,222 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,223 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,223 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,237 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,256 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,256 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,263 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,267 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,283 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,284 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,284 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,297 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,317 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.135:  63%|█████   | 6272/10000 [00:01<00:00, 3746.78samples/s][DEBUG 2017-03-01 15:24:39,317 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,325 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,328 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,347 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,348 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,349 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,358 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,384 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,384 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,392 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,396 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,416 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,418 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,418 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,427 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,449 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.119:  68%|█████▍  | 6784/10000 [00:01<00:00, 3780.45samples/s][DEBUG 2017-03-01 15:24:39,451 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,451 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,469 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,486 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,487 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,487 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,501 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,519 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,519 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,527 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,532 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,551 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,552 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,552 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,566 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,582 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.104:  73%|█████▊  | 7296/10000 [00:02<00:00, 3803.34samples/s][DEBUG 2017-03-01 15:24:39,583 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,583 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,592 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,615 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,616 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,616 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,624 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,646 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,647 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,647 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,659 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,678 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,679 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,679 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,689 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,712 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.092:  78%|██████▏ | 7808/10000 [00:02<00:00, 3847.20samples/s][DEBUG 2017-03-01 15:24:39,712 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,712 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,724 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,743 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,743 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,743 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,754 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,777 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,778 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,778 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,779 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,820 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,821 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,821 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,833 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,852 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.078:  83%|██████▋ | 8320/10000 [00:02<00:00, 3789.42samples/s][DEBUG 2017-03-01 15:24:39,852 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,852 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,864 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,880 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,881 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,881 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,894 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,915 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,916 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,916 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,917 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,949 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:39,949 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,950 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:39,965 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:39,987 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.067:  88%|███████ | 8832/10000 [00:02<00:00, 3785.63samples/s][DEBUG 2017-03-01 15:24:39,988 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:39,988 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,001 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,022 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:40,023 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:40,024 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,037 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,055 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:40,056 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:40,067 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,070 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,089 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.058:  92%|███████▎| 9216/10000 [00:02<00:00, 3775.74samples/s][DEBUG 2017-03-01 15:24:40,090 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:40,090 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,103 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,121 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:40,121 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:40,121 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,134 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,153 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:40,153 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:40,154 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,168 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,190 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.052:  96%|███████▋| 9600/10000 [00:02<00:00, 3788.37samples/s][DEBUG 2017-03-01 15:24:40,191 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:40,191 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,205 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,227 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:40,227 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:40,238 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,242 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,263 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-01 15:24:40,264 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:40,264 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,265 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,300 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.043: 100%|███████▉| 9984/10000 [00:02<00:00, 3695.83samples/s][DEBUG 2017-03-01 15:24:40,301 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-01 15:24:40,309 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.042: 100%|███████| 10000/10000 [00:02<00:00, 3528.21samples/s]
[INFO 2017-03-01 15:24:40,310 kur.model.executor:464] Training loss: 2.042
[INFO 2017-03-01 15:24:40,310 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
[DEBUG 2017-03-01 15:24:40,310 kur.model.executor:430] Saving weights to: t1_dp0.25/cifar.best.train.w
[DEBUG 2017-03-01 15:24:40,310 kur.model.model:213] Saving model weights to: t1_dp0.25/cifar.best.train.w
[DEBUG 2017-03-01 15:24:40,316 kur.loggers.binary_logger:135] Adding data to binary column: training_loss_total
[DEBUG 2017-03-01 15:24:40,317 kur.loggers.binary_logger:135] Adding data to binary column: training_loss_labels
[DEBUG 2017-03-01 15:24:40,318 kur.loggers.binary_logger:135] Adding data to binary column: training_loss_batch
[DEBUG 2017-03-01 15:24:40,319 kur.loggers.binary_logger:144] Writing logger summary.
[DEBUG 2017-03-01 15:24:40,320 kur.loggers.binary_logger:135] Adding data to binary column: batch_loss_total
[DEBUG 2017-03-01 15:24:40,321 kur.loggers.binary_logger:135] Adding data to binary column: batch_loss_labels
[DEBUG 2017-03-01 15:24:40,322 kur.loggers.binary_logger:135] Adding data to binary column: batch_loss_batch
[DEBUG 2017-03-01 15:24:40,323 kur.loggers.binary_logger:144] Writing logger summary.
[DEBUG 2017-03-01 15:24:40,325 kur.model.executor:101] Recompiling the model.
[DEBUG 2017-03-01 15:24:40,327 kur.backend.keras_backend:543] Reusing an existing model.
[DEBUG 2017-03-01 15:24:40,327 kur.backend.keras_backend:560] Assembling a testing function from the model.
[DEBUG 2017-03-01 15:24:40,333 kur.backend.keras_backend:509] Adding additional inputs: labels
[DEBUG 2017-03-01 15:24:40,707 kur.backend.keras_backend:599] Additional inputs for log functions: labels
[DEBUG 2017-03-01 15:24:40,707 kur.backend.keras_backend:616] Expected input shapes: images=(None, 32, 32, 3), labels=(None, None)
[DEBUG 2017-03-01 15:24:40,707 kur.backend.keras_backend:634] Compiled model: {'names': {'output': ['labels', 'labels'], 'input': ['images', 'labels']}, 'func': <keras.backend.theano_backend.Function object at 0x10c5e4320>, 'shapes': {'input': [(None, 32, 32, 3), (None, None)]}}
[DEBUG 2017-03-01 15:24:40,707 kur.providers.batch_provider:57] Batch size set to: 2
[DEBUG 2017-03-01 15:24:40,708 kur.providers.batch_provider:102] Maximum number of batches set to: 1
[INFO 2017-03-01 15:24:40,713 kur.backend.keras_backend:666] Waiting for model to finish compiling...
[DEBUG 2017-03-01 15:24:40,713 kur.providers.batch_provider:139] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,713 kur.providers.batch_provider:204] Next batch of data has been prepared.
Validating, loss=N/A:   0%|                       | 0/6400 [00:00<?, ?samples/s][DEBUG 2017-03-01 15:24:40,717 kur.providers.shuffle_provider:184] Shuffling...
[DEBUG 2017-03-01 15:24:40,853 kur.providers.batch_provider:139] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,854 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,855 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,871 kur.providers.batch_provider:204] Next batch of data has been prepared.
Validating, loss=1.848:   2%|▏         | 128/6400 [00:00<00:07, 821.70samples/s][DEBUG 2017-03-01 15:24:40,873 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,874 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,887 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,900 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,901 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,909 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,914 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,922 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,925 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,933 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,943 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,947 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,948 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,959 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,966 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,969 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,970 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,981 kur.providers.batch_provider:204] Next batch of data has been prepared.
Validating, loss=1.787:  20%|█▌      | 1280/6400 [00:00<00:04, 1135.88samples/s][DEBUG 2017-03-01 15:24:40,983 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:40,992 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:40,997 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,005 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,009 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,021 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,022 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,030 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,035 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,045 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,055 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,060 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,070 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,075 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,075 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,087 kur.providers.batch_provider:204] Next batch of data has been prepared.
Validating, loss=1.799:  36%|██▉     | 2304/6400 [00:00<00:02, 1544.41samples/s][DEBUG 2017-03-01 15:24:41,097 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,102 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,102 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,114 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,119 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,129 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,132 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,145 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,147 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,160 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,160 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,172 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,182 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,188 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,189 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,202 kur.providers.batch_provider:204] Next batch of data has been prepared.
Validating, loss=1.809:  52%|████▏   | 3328/6400 [00:00<00:01, 2055.80samples/s][DEBUG 2017-03-01 15:24:41,203 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,213 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,228 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,233 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,244 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,249 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,250 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,256 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,261 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,275 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,283 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,287 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,288 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,297 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,302 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,311 kur.providers.batch_provider:204] Next batch of data has been prepared.
Validating, loss=1.811:  68%|█████▍  | 4352/6400 [00:00<00:00, 2677.92samples/s][DEBUG 2017-03-01 15:24:41,323 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,328 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,328 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,337 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,342 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,348 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,353 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,362 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,364 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,373 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,378 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,389 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,391 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,399 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,402 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,412 kur.providers.batch_provider:204] Next batch of data has been prepared.
Validating, loss=1.814:  84%|██████▋ | 5376/6400 [00:00<00:00, 3434.82samples/s][DEBUG 2017-03-01 15:24:41,416 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,426 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,431 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,440 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,443 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,453 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,458 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,465 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,469 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,484 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,485 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,492 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-01 15:24:41,497 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-01 15:24:41,504 kur.providers.batch_provider:204] Next batch of data has been prepared.
Validating, loss=1.815: 100%|████████| 6400/6400 [00:00<00:00, 4241.72samples/s]
[INFO 2017-03-01 15:24:41,525 kur.model.executor:197] Validation loss: 1.815
[INFO 2017-03-01 15:24:41,526 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w
[DEBUG 2017-03-01 15:24:41,526 kur.model.executor:444] Copying weights from: t1_dp0.25/cifar.best.train.w
[DEBUG 2017-03-01 15:24:41,529 kur.loggers.binary_logger:135] Adding data to binary column: validation_loss_total
[DEBUG 2017-03-01 15:24:41,530 kur.loggers.binary_logger:135] Adding data to binary column: validation_loss_labels
[DEBUG 2017-03-01 15:24:41,531 kur.loggers.binary_logger:135] Adding data to binary column: validation_loss_batch
[DEBUG 2017-03-01 15:24:41,533 kur.loggers.binary_logger:144] Writing logger summary.
Completed 1 epochs.
[INFO 2017-03-01 15:24:41,537 kur.model.executor:235] Saving most recent weights: t1_dp0.25/cifar.last.w
[DEBUG 2017-03-01 15:24:41,537 kur.model.model:213] Saving model weights to: t1_dp0.25/cifar.last.w

Simplify Kurfile

In [3]:
%%writefile dlnd_p2_dropout.yml

---
settings:

  # Where to get the data
  cifar: &cifar
    url: "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
    checksum: "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce"
    path: "~/kur"           # if kur does not have normalization or one-hot-encoding, ##################
                            # without normalization and one-hot-encoding, the performance will be hurt, right? ####
  # Backend to use                                      
  backend:
    name: keras

  # Hyperparameters
#   cnn:
#     kernels: 20
#     size: [5, 5]
#     strides: [2, 2]                

#   pool:
#     size: [2,2]
#     strides: [2,2]
#     type: max                         

model:
  - input: images                       
  - convolution:                       
      kernels: 20
      size: [5, 5]
      strides: [2,2]
  - activation: relu
  - pool:
      size: [2,2]
      strides: [2,2]
      type: max
  - flatten:
  - dense: 15                           
  - dropout: 0.25
  - dense: 10
  - activation: softmax
    name: labels

train:
  data:
    - cifar:
        <<: *cifar
        parts: [1]                      
  provider: &provider
    batch_size: 128
    num_batches: 3                        # use all batches
    randomize: True
  log: t1_dp0.25/cifar-log                   
  epochs: 1
  weights:
    initial: t1_dp0.25/cifar.best.valid.w    
    best: t1_dp0.25/cifar.best.train.w
    last: t1_dp0.25/cifar.last.w

  optimizer:
    name: adam
    learning_rate: 0.001

validate:
  data:
    - cifar:
       <<: *cifar
       parts: 5                          
  provider: 
    <<: *provider       # can I do this? ##########
    num_batches: 2     # override num_batches to be 50 while keep batch_size and randomize as 128 and True ####
  weights: t1_dp0.25/cifar.best.valid.w

test: &test
  data:
    - cifar:
       <<: *cifar
       parts: test
  weights: t1_dp0.25/cifar.best.valid.w
  provider: 
    <<: *provider
    randomize: False


evaluate:
  <<: *test
  destination: t1_dp0.25/cifar.results.pkl

loss:
  - target: labels                        # in the project: training loss and valid_accuracy are printed #############
    name: categorical_crossentropy        # this should be a matter of personal taste, won't really affect anything##
...
Overwriting dlnd_p2_dropout.yml
In [4]:
!kur -vv train dlnd_p2_dropout.yml
[INFO 2017-03-02 20:31:03,897 kur.kurfile:699] Parsing source: dlnd_p2_dropout.yml, included by top-level.
[INFO 2017-03-02 20:31:03,910 kur.kurfile:82] Parsing Kurfile...
[DEBUG 2017-03-02 20:31:03,910 kur.kurfile:784] Parsing Kurfile section: settings
[DEBUG 2017-03-02 20:31:03,914 kur.kurfile:784] Parsing Kurfile section: train
[DEBUG 2017-03-02 20:31:03,917 kur.kurfile:784] Parsing Kurfile section: validate
[DEBUG 2017-03-02 20:31:03,919 kur.kurfile:784] Parsing Kurfile section: test
[DEBUG 2017-03-02 20:31:03,921 kur.kurfile:784] Parsing Kurfile section: evaluate
[DEBUG 2017-03-02 20:31:03,925 kur.containers.layers.placeholder:63] Using short-hand name for placeholder: images
[DEBUG 2017-03-02 20:31:03,925 kur.containers.layers.placeholder:97] Placeholder "images" has a deferred shape.
[DEBUG 2017-03-02 20:31:03,927 kur.kurfile:784] Parsing Kurfile section: loss
[INFO 2017-03-02 20:31:03,929 kur.loggers.binary_logger:107] Log does not exist. Creating path: t1_dp0.25/cifar-log
[DEBUG 2017-03-02 20:31:04,727 kur.utils.package:233] File exists and passed checksum: /Users/Natsume/kur/cifar-10-python.tar.gz
[DEBUG 2017-03-02 20:31:11,034 kur.providers.batch_provider:57] Batch size set to: 128
[DEBUG 2017-03-02 20:31:11,034 kur.providers.batch_provider:102] Maximum number of batches set to: 3
[DEBUG 2017-03-02 20:31:11,766 kur.utils.package:233] File exists and passed checksum: /Users/Natsume/kur/cifar-10-python.tar.gz
[DEBUG 2017-03-02 20:31:17,439 kur.providers.batch_provider:57] Batch size set to: 128
[DEBUG 2017-03-02 20:31:17,439 kur.providers.batch_provider:102] Maximum number of batches set to: 2
[DEBUG 2017-03-02 20:31:17,439 kur.backend.backend:187] Using backend: keras
[INFO 2017-03-02 20:31:17,440 kur.backend.backend:80] Creating backend: keras
[INFO 2017-03-02 20:31:17,440 kur.backend.backend:83] Backend variants: none
[INFO 2017-03-02 20:31:17,440 kur.backend.keras_backend:122] No particular backend for Keras has been requested.
[DEBUG 2017-03-02 20:31:17,440 kur.backend.keras_backend:124] Using the system-default Keras backend.
[DEBUG 2017-03-02 20:31:17,440 kur.backend.keras_backend:189] Overriding environmental variables: {'THEANO_FLAGS': None, 'TF_CPP_MIN_LOG_LEVEL': '1', 'KERAS_BACKEND': None}
[INFO 2017-03-02 20:31:18,398 kur.backend.keras_backend:195] Keras is loaded. The backend is: theano
[INFO 2017-03-02 20:31:18,399 kur.model.model:260] Enumerating the model containers.
[INFO 2017-03-02 20:31:18,399 kur.model.model:265] Assembling the model dependency graph.
[DEBUG 2017-03-02 20:31:18,400 kur.model.model:272] Assembled Node: images
[DEBUG 2017-03-02 20:31:18,400 kur.model.model:274]   Uses: 
[DEBUG 2017-03-02 20:31:18,400 kur.model.model:276]   Used by: ..convolution.0
[DEBUG 2017-03-02 20:31:18,400 kur.model.model:277]   Aliases: images
[DEBUG 2017-03-02 20:31:18,400 kur.model.model:272] Assembled Node: ..convolution.0
[DEBUG 2017-03-02 20:31:18,400 kur.model.model:274]   Uses: images
[DEBUG 2017-03-02 20:31:18,400 kur.model.model:276]   Used by: ..activation.0
[DEBUG 2017-03-02 20:31:18,400 kur.model.model:277]   Aliases: ..convolution.0
[DEBUG 2017-03-02 20:31:18,400 kur.model.model:272] Assembled Node: ..activation.0
[DEBUG 2017-03-02 20:31:18,400 kur.model.model:274]   Uses: ..convolution.0
[DEBUG 2017-03-02 20:31:18,400 kur.model.model:276]   Used by: ..pool.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:277]   Aliases: ..activation.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:272] Assembled Node: ..pool.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:274]   Uses: ..activation.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:276]   Used by: ..flatten.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:277]   Aliases: ..pool.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:272] Assembled Node: ..flatten.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:274]   Uses: ..pool.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:276]   Used by: ..dense.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:277]   Aliases: ..flatten.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:272] Assembled Node: ..dense.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:274]   Uses: ..flatten.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:276]   Used by: ..dropout.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:277]   Aliases: ..dense.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:272] Assembled Node: ..dropout.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:274]   Uses: ..dense.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:276]   Used by: ..dense.1
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:277]   Aliases: ..dropout.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:272] Assembled Node: ..dense.1
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:274]   Uses: ..dropout.0
[DEBUG 2017-03-02 20:31:18,401 kur.model.model:276]   Used by: labels
[DEBUG 2017-03-02 20:31:18,402 kur.model.model:277]   Aliases: ..dense.1
[DEBUG 2017-03-02 20:31:18,402 kur.model.model:272] Assembled Node: labels
[DEBUG 2017-03-02 20:31:18,402 kur.model.model:274]   Uses: ..dense.1
[DEBUG 2017-03-02 20:31:18,402 kur.model.model:276]   Used by: 
[DEBUG 2017-03-02 20:31:18,402 kur.model.model:277]   Aliases: labels
[INFO 2017-03-02 20:31:18,402 kur.model.model:280] Connecting the model graph.
[DEBUG 2017-03-02 20:31:18,402 kur.model.model:311] Building node: images
[DEBUG 2017-03-02 20:31:18,402 kur.model.model:312]   Aliases: images
[DEBUG 2017-03-02 20:31:18,402 kur.model.model:313]   Inputs:
[DEBUG 2017-03-02 20:31:18,402 kur.containers.layers.placeholder:117] Creating placeholder for "images" with data type "float32".
[DEBUG 2017-03-02 20:31:18,402 kur.model.model:125] Trying to infer shape for input "images"
[DEBUG 2017-03-02 20:31:18,402 kur.model.model:143] Inferred shape for input "images": (32, 32, 3)
[DEBUG 2017-03-02 20:31:18,402 kur.containers.layers.placeholder:127] Inferred shape: (32, 32, 3)
[DEBUG 2017-03-02 20:31:18,408 kur.model.model:382]   Value: images
[DEBUG 2017-03-02 20:31:18,408 kur.model.model:311] Building node: ..convolution.0
[DEBUG 2017-03-02 20:31:18,408 kur.model.model:312]   Aliases: ..convolution.0
[DEBUG 2017-03-02 20:31:18,408 kur.model.model:313]   Inputs:
[DEBUG 2017-03-02 20:31:18,408 kur.model.model:315]   - images: images
[DEBUG 2017-03-02 20:31:19,584 kur.model.model:382]   Value: Elemwise{add,no_inplace}.0
[DEBUG 2017-03-02 20:31:19,585 kur.model.model:311] Building node: ..activation.0
[DEBUG 2017-03-02 20:31:19,585 kur.model.model:312]   Aliases: ..activation.0
[DEBUG 2017-03-02 20:31:19,585 kur.model.model:313]   Inputs:
[DEBUG 2017-03-02 20:31:19,585 kur.model.model:315]   - ..convolution.0: Elemwise{add,no_inplace}.0
[DEBUG 2017-03-02 20:31:19,623 kur.model.model:382]   Value: Elemwise{mul,no_inplace}.0
[DEBUG 2017-03-02 20:31:19,623 kur.model.model:311] Building node: ..pool.0
[DEBUG 2017-03-02 20:31:19,623 kur.model.model:312]   Aliases: ..pool.0
[DEBUG 2017-03-02 20:31:19,623 kur.model.model:313]   Inputs:
[DEBUG 2017-03-02 20:31:19,623 kur.model.model:315]   - ..activation.0: Elemwise{mul,no_inplace}.0
[DEBUG 2017-03-02 20:31:19,626 kur.model.model:382]   Value: DimShuffle{0,2,3,1}.0
[DEBUG 2017-03-02 20:31:19,626 kur.model.model:311] Building node: ..flatten.0
[DEBUG 2017-03-02 20:31:19,626 kur.model.model:312]   Aliases: ..flatten.0
[DEBUG 2017-03-02 20:31:19,627 kur.model.model:313]   Inputs:
[DEBUG 2017-03-02 20:31:19,627 kur.model.model:315]   - ..pool.0: DimShuffle{0,2,3,1}.0
[DEBUG 2017-03-02 20:31:19,636 kur.model.model:382]   Value: Reshape{2}.0
[DEBUG 2017-03-02 20:31:19,636 kur.model.model:311] Building node: ..dense.0
[DEBUG 2017-03-02 20:31:19,636 kur.model.model:312]   Aliases: ..dense.0
[DEBUG 2017-03-02 20:31:19,636 kur.model.model:313]   Inputs:
[DEBUG 2017-03-02 20:31:19,636 kur.model.model:315]   - ..flatten.0: Reshape{2}.0
[DEBUG 2017-03-02 20:31:19,638 kur.model.model:382]   Value: Elemwise{add,no_inplace}.0
[DEBUG 2017-03-02 20:31:19,638 kur.model.model:311] Building node: ..dropout.0
[DEBUG 2017-03-02 20:31:19,639 kur.model.model:312]   Aliases: ..dropout.0
[DEBUG 2017-03-02 20:31:19,639 kur.model.model:313]   Inputs:
[DEBUG 2017-03-02 20:31:19,639 kur.model.model:315]   - ..dense.0: Elemwise{add,no_inplace}.0
[DEBUG 2017-03-02 20:31:19,752 kur.model.model:382]   Value: if{}.0
[DEBUG 2017-03-02 20:31:19,752 kur.model.model:311] Building node: ..dense.1
[DEBUG 2017-03-02 20:31:19,752 kur.model.model:312]   Aliases: ..dense.1
[DEBUG 2017-03-02 20:31:19,752 kur.model.model:313]   Inputs:
[DEBUG 2017-03-02 20:31:19,752 kur.model.model:315]   - ..dropout.0: if{}.0
[DEBUG 2017-03-02 20:31:19,753 kur.model.model:382]   Value: Elemwise{add,no_inplace}.0
[DEBUG 2017-03-02 20:31:19,753 kur.model.model:311] Building node: labels
[DEBUG 2017-03-02 20:31:19,754 kur.model.model:312]   Aliases: labels
[DEBUG 2017-03-02 20:31:19,754 kur.model.model:313]   Inputs:
[DEBUG 2017-03-02 20:31:19,754 kur.model.model:315]   - ..dense.1: Elemwise{add,no_inplace}.0
[DEBUG 2017-03-02 20:31:19,754 kur.model.model:382]   Value: Softmax.0
[INFO 2017-03-02 20:31:19,754 kur.model.model:284] Model inputs:  images
[INFO 2017-03-02 20:31:19,754 kur.model.model:285] Model outputs: labels
[INFO 2017-03-02 20:31:19,754 kur.kurfile:357] Ignoring missing initial weights: t1_dp0.25/cifar.best.valid.w. If this is undesireable, set "must_exist" to "yes" in the approriate "weights" section.
[INFO 2017-03-02 20:31:19,754 kur.model.executor:315] No historical training loss available from logs.
[INFO 2017-03-02 20:31:19,755 kur.model.executor:323] No historical validation loss available from logs.
[INFO 2017-03-02 20:31:19,755 kur.model.executor:329] No previous epochs.
[DEBUG 2017-03-02 20:31:19,755 kur.model.executor:353] Epoch handling mode: additional
[DEBUG 2017-03-02 20:31:19,755 kur.model.executor:101] Recompiling the model.
[DEBUG 2017-03-02 20:31:19,755 kur.backend.keras_backend:527] Instantiating a Keras model.
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] Layer (type)                     Output Shape          Param #     Connected to                     
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ====================================================================================================
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] images (InputLayer)              (None, 32, 32, 3)     0                                            
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ..convolution.0 (Convolution2D)  (None, 16, 16, 20)    1520        images[0][0]                     
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ..activation.0 (Activation)      (None, 16, 16, 20)    0           ..convolution.0[0][0]            
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ..pool.0 (MaxPooling2D)          (None, 8, 8, 20)      0           ..activation.0[0][0]             
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ..flatten.0 (Flatten)            (None, 1280)          0           ..pool.0[0][0]                   
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ..dense.0 (Dense)                (None, 15)            19215       ..flatten.0[0][0]                
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ..dropout.0 (Dropout)            (None, 15)            0           ..dense.0[0][0]                  
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ..dense.1 (Dense)                (None, 10)            160         ..dropout.0[0][0]                
[DEBUG 2017-03-02 20:31:20,003 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-02 20:31:20,004 kur.backend.keras_backend:538] labels (Activation)              (None, 10)            0           ..dense.1[0][0]                  
[DEBUG 2017-03-02 20:31:20,004 kur.backend.keras_backend:538] ====================================================================================================
[DEBUG 2017-03-02 20:31:20,004 kur.backend.keras_backend:538] Total params: 20,895
[DEBUG 2017-03-02 20:31:20,004 kur.backend.keras_backend:538] Trainable params: 20,895
[DEBUG 2017-03-02 20:31:20,004 kur.backend.keras_backend:538] Non-trainable params: 0
[DEBUG 2017-03-02 20:31:20,004 kur.backend.keras_backend:538] ____________________________________________________________________________________________________
[DEBUG 2017-03-02 20:31:20,004 kur.backend.keras_backend:538] 
[DEBUG 2017-03-02 20:31:20,004 kur.backend.keras_backend:576] Assembling a training function from the model.
[DEBUG 2017-03-02 20:31:20,011 kur.backend.keras_backend:509] Adding additional inputs: labels
[DEBUG 2017-03-02 20:31:21,492 kur.backend.keras_backend:599] Additional inputs for log functions: labels
[DEBUG 2017-03-02 20:31:21,492 kur.backend.keras_backend:616] Expected input shapes: images=(None, 32, 32, 3), labels=(None, None)
[DEBUG 2017-03-02 20:31:21,492 kur.backend.keras_backend:634] Compiled model: {'names': {'output': ['labels', 'labels'], 'input': ['images', 'labels']}, 'shapes': {'input': [(None, 32, 32, 3), (None, None)]}, 'func': <keras.backend.theano_backend.Function object at 0x11f7def28>}
[DEBUG 2017-03-02 20:31:21,493 kur.providers.batch_provider:57] Batch size set to: 2
[DEBUG 2017-03-02 20:31:21,493 kur.providers.batch_provider:102] Maximum number of batches set to: 1
[INFO 2017-03-02 20:31:21,497 kur.backend.keras_backend:666] Waiting for model to finish compiling...
[DEBUG 2017-03-02 20:31:21,497 kur.providers.batch_provider:139] Preparing next batch of data...
[DEBUG 2017-03-02 20:31:21,497 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-02 20:31:21,551 kur.providers.provider:144] Data source "images": entries=10000, shape=(32, 32, 3)
[DEBUG 2017-03-02 20:31:21,551 kur.providers.provider:144] Data source "labels": entries=10000, shape=(10,)

Epoch 1/1, loss=N/A:   0%|                         | 0/384 [00:00<?, ?samples/s][DEBUG 2017-03-02 20:31:21,565 kur.providers.shuffle_provider:184] Shuffling...
[DEBUG 2017-03-02 20:31:21,855 kur.providers.batch_provider:139] Preparing next batch of data...
[DEBUG 2017-03-02 20:31:21,856 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-02 20:31:21,856 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-02 20:31:21,865 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-02 20:31:21,868 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-02 20:31:21,894 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-02 20:31:21,894 kur.loggers.binary_logger:135] Adding data to binary column: batch_loss_batch
[DEBUG 2017-03-02 20:31:21,895 kur.loggers.binary_logger:135] Adding data to binary column: batch_loss_total
[DEBUG 2017-03-02 20:31:21,895 kur.loggers.binary_logger:135] Adding data to binary column: batch_loss_labels
[DEBUG 2017-03-02 20:31:21,896 kur.loggers.binary_logger:144] Writing logger summary.
Epoch 1/1, loss=2.316:  33%|████        | 128/384 [00:00<00:00, 384.88samples/s][DEBUG 2017-03-02 20:31:21,898 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-02 20:31:21,898 kur.providers.batch_provider:156] Preparing next batch of data...
[DEBUG 2017-03-02 20:31:21,911 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-02 20:31:21,936 kur.model.executor:597] Finished training on batch.
[DEBUG 2017-03-02 20:31:21,937 kur.model.executor:578] Training on batch...
[DEBUG 2017-03-02 20:31:21,964 kur.model.executor:597] Finished training on batch.
Epoch 1/1, loss=2.300: 100%|████████████| 384/384 [00:00<00:00, 962.01samples/s]
[INFO 2017-03-02 20:31:21,964 kur.model.executor:464] Training loss: 2.300
[INFO 2017-03-02 20:31:21,965 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
[DEBUG 2017-03-02 20:31:21,965 kur.model.executor:430] Saving weights to: t1_dp0.25/cifar.best.train.w
[DEBUG 2017-03-02 20:31:21,965 kur.model.model:213] Saving model weights to: t1_dp0.25/cifar.best.train.w
[DEBUG 2017-03-02 20:31:21,969 kur.loggers.binary_logger:135] Adding data to binary column: batch_loss_batch
[DEBUG 2017-03-02 20:31:21,970 kur.loggers.binary_logger:135] Adding data to binary column: batch_loss_total
[DEBUG 2017-03-02 20:31:21,970 kur.loggers.binary_logger:135] Adding data to binary column: batch_loss_labels
[DEBUG 2017-03-02 20:31:21,970 kur.loggers.binary_logger:144] Writing logger summary.
[DEBUG 2017-03-02 20:31:21,971 kur.loggers.binary_logger:135] Adding data to binary column: training_loss_batch
[DEBUG 2017-03-02 20:31:21,972 kur.loggers.binary_logger:135] Adding data to binary column: training_loss_total
[DEBUG 2017-03-02 20:31:21,973 kur.loggers.binary_logger:135] Adding data to binary column: training_loss_labels
[DEBUG 2017-03-02 20:31:21,973 kur.loggers.binary_logger:144] Writing logger summary.
[DEBUG 2017-03-02 20:31:21,974 kur.model.executor:101] Recompiling the model.
[DEBUG 2017-03-02 20:31:21,975 kur.backend.keras_backend:543] Reusing an existing model.
[DEBUG 2017-03-02 20:31:21,975 kur.backend.keras_backend:560] Assembling a testing function from the model.
[DEBUG 2017-03-02 20:31:21,982 kur.backend.keras_backend:509] Adding additional inputs: labels
[DEBUG 2017-03-02 20:31:22,340 kur.backend.keras_backend:599] Additional inputs for log functions: labels
[DEBUG 2017-03-02 20:31:22,340 kur.backend.keras_backend:616] Expected input shapes: images=(None, 32, 32, 3), labels=(None, None)
[DEBUG 2017-03-02 20:31:22,340 kur.backend.keras_backend:634] Compiled model: {'names': {'output': ['labels', 'labels'], 'input': ['images', 'labels']}, 'shapes': {'input': [(None, 32, 32, 3), (None, None)]}, 'func': <keras.backend.theano_backend.Function object at 0x11fc74400>}
[DEBUG 2017-03-02 20:31:22,340 kur.providers.batch_provider:57] Batch size set to: 2
[DEBUG 2017-03-02 20:31:22,341 kur.providers.batch_provider:102] Maximum number of batches set to: 1
[INFO 2017-03-02 20:31:22,343 kur.backend.keras_backend:666] Waiting for model to finish compiling...
[DEBUG 2017-03-02 20:31:22,343 kur.providers.batch_provider:139] Preparing next batch of data...
[DEBUG 2017-03-02 20:31:22,343 kur.providers.batch_provider:204] Next batch of data has been prepared.
Validating, loss=N/A:   0%|                        | 0/256 [00:00<?, ?samples/s][DEBUG 2017-03-02 20:31:22,346 kur.providers.shuffle_provider:184] Shuffling...
[DEBUG 2017-03-02 20:31:22,612 kur.providers.batch_provider:139] Preparing next batch of data...
[DEBUG 2017-03-02 20:31:22,613 kur.providers.batch_provider:204] Next batch of data has been prepared.
[DEBUG 2017-03-02 20:31:22,619 kur.providers.batch_provider:156] Preparing next batch of data...
Validating, loss=2.300:  50%|█████▌     | 128/256 [00:00<00:00, 463.55samples/s][DEBUG 2017-03-02 20:31:22,622 kur.providers.batch_provider:204] Next batch of data has been prepared.
Validating, loss=2.290: 100%|███████████| 256/256 [00:00<00:00, 900.46samples/s]
[INFO 2017-03-02 20:31:22,631 kur.model.executor:197] Validation loss: 2.290
[INFO 2017-03-02 20:31:22,631 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w
[DEBUG 2017-03-02 20:31:22,631 kur.model.executor:444] Copying weights from: t1_dp0.25/cifar.best.train.w
[DEBUG 2017-03-02 20:31:22,633 kur.loggers.binary_logger:135] Adding data to binary column: validation_loss_batch
[DEBUG 2017-03-02 20:31:22,633 kur.loggers.binary_logger:135] Adding data to binary column: validation_loss_total
[DEBUG 2017-03-02 20:31:22,634 kur.loggers.binary_logger:135] Adding data to binary column: validation_loss_labels
[DEBUG 2017-03-02 20:31:22,634 kur.loggers.binary_logger:144] Writing logger summary.
Completed 1 epochs.
[INFO 2017-03-02 20:31:22,635 kur.model.executor:235] Saving most recent weights: t1_dp0.25/cifar.last.w
[DEBUG 2017-03-02 20:31:22,636 kur.model.model:213] Saving model weights to: t1_dp0.25/cifar.last.w
In [1]:
%%writefile dlnd_p2_dropout.yml

---
settings:

  # Where to get the data
  cifar: &cifar
    url: "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
    checksum: "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce"
    path: "~/kur"           # if kur does not have normalization or one-hot-encoding, ##################
                            # without normalization and one-hot-encoding, the performance will be hurt, right? ####
  # Backend to use                                      
  backend:
    name: keras

  # Hyperparameters
#   cnn:
#     kernels: 20
#     size: [5, 5]
#     strides: [2, 2]                

#   pool:
#     size: [2,2]
#     strides: [2,2]
#     type: max                         

model:
  - input: images                       
  - convolution:                       
      kernels: 20
      size: [5, 5]
      strides: [2,2]
  - activation: relu
  - pool:
      size: [2,2]
      strides: [2,2]
      type: max
  - flatten:
  - dense: 15                           
  - dropout: 0.25
  - dense: 10
  - activation: softmax
    name: labels

train:
  data:
    - cifar:
        <<: *cifar
        parts: [1]                      
  provider: &provider
    batch_size: 128
    num_batches: 1000                        # use all batches
    randomize: True
  log: t1_dp0.25/cifar-log                   
  epochs: 20
  weights:
    initial: t1_dp0.25/cifar.best.valid.w    
    best: t1_dp0.25/cifar.best.train.w
    last: t1_dp0.25/cifar.last.w

  optimizer:
    name: adam
    learning_rate: 0.001
        
    #  hooks:
    #    - plot: loss.png

  hooks:
    - plot:
        loss_per_batch: t1_dp0.25/loss1.png
        loss_per_time: t1_dp0.25/loss2.png
        throughput_per_time: t1_dp0.25/loss3.png

validate:
  data:
    - cifar:
       <<: *cifar
       parts: 5                          
  provider: 
    <<: *provider       
    num_batches: 50     
  weights: t1_dp0.25/cifar.best.valid.w

test: &test
  data:
    - cifar:
       <<: *cifar
       parts: test
  weights: t1_dp0.25/cifar.best.valid.w
  provider: 
    <<: *provider
    randomize: False


evaluate:
  <<: *test
  destination: t1_dp0.25/cifar.results.pkl

loss:
  - target: labels                        
    name: categorical_crossentropy        
        

...
Overwriting dlnd_p2_dropout.yml
In [3]:
!kur -v train dlnd_p2_dropout.yml
[INFO 2017-03-02 22:41:19,495 kur.kurfile:699] Parsing source: dlnd_p2_dropout.yml, included by top-level.
[INFO 2017-03-02 22:41:19,511 kur.kurfile:82] Parsing Kurfile...
[INFO 2017-03-02 22:41:19,532 kur.loggers.binary_logger:107] Log does not exist. Creating path: t1_dp0.25/cifar-log
[INFO 2017-03-02 22:41:36,185 kur.backend.backend:80] Creating backend: keras
[INFO 2017-03-02 22:41:36,185 kur.backend.backend:83] Backend variants: none
[INFO 2017-03-02 22:41:36,185 kur.backend.keras_backend:122] No particular backend for Keras has been requested.
[INFO 2017-03-02 22:41:37,112 kur.backend.keras_backend:195] Keras is loaded. The backend is: theano
[INFO 2017-03-02 22:41:37,113 kur.model.model:260] Enumerating the model containers.
[INFO 2017-03-02 22:41:37,113 kur.model.model:265] Assembling the model dependency graph.
[INFO 2017-03-02 22:41:37,113 kur.model.model:280] Connecting the model graph.
[INFO 2017-03-02 22:41:38,321 kur.model.model:284] Model inputs:  images
[INFO 2017-03-02 22:41:38,321 kur.model.model:285] Model outputs: labels
[INFO 2017-03-02 22:41:38,321 kur.kurfile:357] Ignoring missing initial weights: t1_dp0.25/cifar.best.valid.w. If this is undesireable, set "must_exist" to "yes" in the approriate "weights" section.
[INFO 2017-03-02 22:41:38,322 kur.model.executor:315] No historical training loss available from logs.
[INFO 2017-03-02 22:41:38,322 kur.model.executor:323] No historical validation loss available from logs.
[INFO 2017-03-02 22:41:38,322 kur.model.executor:329] No previous epochs.
[INFO 2017-03-02 22:41:39,765 kur.backend.keras_backend:666] Waiting for model to finish compiling...

Epoch 1/20, loss=2.077: 100%|██████| 10000/10000 [00:02<00:00, 3982.49samples/s]
[INFO 2017-03-02 22:41:42,548 kur.model.executor:464] Training loss: 2.077
[INFO 2017-03-02 22:41:42,548 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
[INFO 2017-03-02 22:41:42,946 kur.backend.keras_backend:666] Waiting for model to finish compiling...
Validating, loss=1.857: 100%|████████| 6400/6400 [00:01<00:00, 5417.04samples/s]
[INFO 2017-03-02 22:41:44,131 kur.model.executor:197] Validation loss: 1.857
[INFO 2017-03-02 22:41:44,132 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 2/20, loss=1.788: 100%|██████| 10000/10000 [00:02<00:00, 4656.73samples/s]
[INFO 2017-03-02 22:41:47,586 kur.model.executor:464] Training loss: 1.788
[INFO 2017-03-02 22:41:47,586 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.659: 100%|████████| 6400/6400 [00:00<00:00, 9786.11samples/s]
[INFO 2017-03-02 22:41:48,252 kur.model.executor:197] Validation loss: 1.659
[INFO 2017-03-02 22:41:48,252 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 3/20, loss=1.646: 100%|██████| 10000/10000 [00:02<00:00, 4510.26samples/s]
[INFO 2017-03-02 22:41:51,538 kur.model.executor:464] Training loss: 1.646
[INFO 2017-03-02 22:41:51,538 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.541: 100%|████████| 6400/6400 [00:00<00:00, 9282.69samples/s]
[INFO 2017-03-02 22:41:52,243 kur.model.executor:197] Validation loss: 1.541
[INFO 2017-03-02 22:41:52,243 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 4/20, loss=1.557: 100%|██████| 10000/10000 [00:02<00:00, 4589.28samples/s]
[INFO 2017-03-02 22:41:55,775 kur.model.executor:464] Training loss: 1.557
[INFO 2017-03-02 22:41:55,775 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.491: 100%|███████| 6400/6400 [00:00<00:00, 10572.41samples/s]
[INFO 2017-03-02 22:41:56,389 kur.model.executor:197] Validation loss: 1.491
[INFO 2017-03-02 22:41:56,390 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 5/20, loss=1.500: 100%|██████| 10000/10000 [00:02<00:00, 4751.18samples/s]
[INFO 2017-03-02 22:41:59,591 kur.model.executor:464] Training loss: 1.500
[INFO 2017-03-02 22:41:59,591 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.454: 100%|███████| 6400/6400 [00:00<00:00, 10827.84samples/s]
[INFO 2017-03-02 22:42:00,192 kur.model.executor:197] Validation loss: 1.454
[INFO 2017-03-02 22:42:00,192 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 6/20, loss=1.462: 100%|██████| 10000/10000 [00:01<00:00, 5120.87samples/s]
[INFO 2017-03-02 22:42:03,203 kur.model.executor:464] Training loss: 1.462
[INFO 2017-03-02 22:42:03,203 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.416: 100%|███████| 6400/6400 [00:00<00:00, 11439.95samples/s]
[INFO 2017-03-02 22:42:03,774 kur.model.executor:197] Validation loss: 1.416
[INFO 2017-03-02 22:42:03,775 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 7/20, loss=1.420: 100%|██████| 10000/10000 [00:01<00:00, 5101.30samples/s]
[INFO 2017-03-02 22:42:06,715 kur.model.executor:464] Training loss: 1.420
[INFO 2017-03-02 22:42:06,715 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.398: 100%|███████| 6400/6400 [00:00<00:00, 11667.38samples/s]
[INFO 2017-03-02 22:42:07,271 kur.model.executor:197] Validation loss: 1.398
[INFO 2017-03-02 22:42:07,272 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 8/20, loss=1.404: 100%|██████| 10000/10000 [00:02<00:00, 4778.85samples/s]
[INFO 2017-03-02 22:42:10,564 kur.model.executor:464] Training loss: 1.404
[INFO 2017-03-02 22:42:10,565 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.383: 100%|███████| 6400/6400 [00:00<00:00, 10729.32samples/s]
[INFO 2017-03-02 22:42:11,171 kur.model.executor:197] Validation loss: 1.383
[INFO 2017-03-02 22:42:11,171 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 9/20, loss=1.358: 100%|██████| 10000/10000 [00:02<00:00, 4984.91samples/s]
[INFO 2017-03-02 22:42:14,238 kur.model.executor:464] Training loss: 1.358
[INFO 2017-03-02 22:42:14,238 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.366: 100%|███████| 6400/6400 [00:00<00:00, 11038.42samples/s]
[INFO 2017-03-02 22:42:14,827 kur.model.executor:197] Validation loss: 1.366
[INFO 2017-03-02 22:42:14,828 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 10/20, loss=1.342: 100%|█████| 10000/10000 [00:01<00:00, 5038.18samples/s]
[INFO 2017-03-02 22:42:17,884 kur.model.executor:464] Training loss: 1.342
[INFO 2017-03-02 22:42:17,885 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.358: 100%|███████| 6400/6400 [00:00<00:00, 10852.61samples/s]
[INFO 2017-03-02 22:42:18,485 kur.model.executor:197] Validation loss: 1.358
[INFO 2017-03-02 22:42:18,486 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 11/20, loss=1.315: 100%|█████| 10000/10000 [00:02<00:00, 4786.46samples/s]
[INFO 2017-03-02 22:42:21,742 kur.model.executor:464] Training loss: 1.315
[INFO 2017-03-02 22:42:21,742 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.337: 100%|███████| 6400/6400 [00:00<00:00, 11502.12samples/s]
[INFO 2017-03-02 22:42:22,308 kur.model.executor:197] Validation loss: 1.337
[INFO 2017-03-02 22:42:22,309 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 12/20, loss=1.285: 100%|█████| 10000/10000 [00:02<00:00, 4919.16samples/s]
[INFO 2017-03-02 22:42:25,371 kur.model.executor:464] Training loss: 1.285
[INFO 2017-03-02 22:42:25,371 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.324: 100%|███████| 6400/6400 [00:00<00:00, 11415.50samples/s]
[INFO 2017-03-02 22:42:25,940 kur.model.executor:197] Validation loss: 1.324
[INFO 2017-03-02 22:42:25,940 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 13/20, loss=1.269: 100%|█████| 10000/10000 [00:02<00:00, 4059.17samples/s]
[INFO 2017-03-02 22:42:29,429 kur.model.executor:464] Training loss: 1.269
[INFO 2017-03-02 22:42:29,429 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.312: 100%|████████| 6400/6400 [00:00<00:00, 7220.12samples/s]
[INFO 2017-03-02 22:42:30,326 kur.model.executor:197] Validation loss: 1.312
[INFO 2017-03-02 22:42:30,326 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 14/20, loss=1.243: 100%|█████| 10000/10000 [00:02<00:00, 4731.76samples/s]
[INFO 2017-03-02 22:42:33,863 kur.model.executor:464] Training loss: 1.243
[INFO 2017-03-02 22:42:33,863 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.307: 100%|███████| 6400/6400 [00:00<00:00, 10456.27samples/s]
[INFO 2017-03-02 22:42:34,483 kur.model.executor:197] Validation loss: 1.307
[INFO 2017-03-02 22:42:34,483 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 15/20, loss=1.240: 100%|█████| 10000/10000 [00:02<00:00, 4682.33samples/s]
[INFO 2017-03-02 22:42:37,757 kur.model.executor:464] Training loss: 1.240
[INFO 2017-03-02 22:42:37,757 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.287: 100%|████████| 6400/6400 [00:00<00:00, 9928.56samples/s]
[INFO 2017-03-02 22:42:38,409 kur.model.executor:197] Validation loss: 1.287
[INFO 2017-03-02 22:42:38,410 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 16/20, loss=1.205: 100%|█████| 10000/10000 [00:02<00:00, 4778.99samples/s]
[INFO 2017-03-02 22:42:41,582 kur.model.executor:464] Training loss: 1.205
[INFO 2017-03-02 22:42:41,582 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.270: 100%|███████| 6400/6400 [00:00<00:00, 11664.23samples/s]
[INFO 2017-03-02 22:42:42,139 kur.model.executor:197] Validation loss: 1.270
[INFO 2017-03-02 22:42:42,139 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 17/20, loss=1.193: 100%|█████| 10000/10000 [00:02<00:00, 4769.72samples/s]
[INFO 2017-03-02 22:42:45,326 kur.model.executor:464] Training loss: 1.193
[INFO 2017-03-02 22:42:45,326 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.271: 100%|███████| 6400/6400 [00:00<00:00, 10762.94samples/s]
[INFO 2017-03-02 22:42:45,928 kur.model.executor:197] Validation loss: 1.271

Epoch 18/20, loss=1.183: 100%|█████| 10000/10000 [00:02<00:00, 3339.14samples/s]
[INFO 2017-03-02 22:42:50,414 kur.model.executor:464] Training loss: 1.183
[INFO 2017-03-02 22:42:50,414 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.289: 100%|████████| 6400/6400 [00:00<00:00, 8447.62samples/s]
[INFO 2017-03-02 22:42:51,181 kur.model.executor:197] Validation loss: 1.289

Epoch 19/20, loss=1.166: 100%|█████| 10000/10000 [00:02<00:00, 3746.48samples/s]
[INFO 2017-03-02 22:42:55,805 kur.model.executor:464] Training loss: 1.166
[INFO 2017-03-02 22:42:55,805 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.258: 100%|████████| 6400/6400 [00:00<00:00, 9802.57samples/s]
[INFO 2017-03-02 22:42:56,466 kur.model.executor:197] Validation loss: 1.258
[INFO 2017-03-02 22:42:56,466 kur.model.executor:413] Saving best historical validation weights: t1_dp0.25/cifar.best.valid.w

Epoch 20/20, loss=1.142: 100%|█████| 10000/10000 [00:02<00:00, 4070.94samples/s]
[INFO 2017-03-02 22:43:00,335 kur.model.executor:464] Training loss: 1.142
[INFO 2017-03-02 22:43:00,335 kur.model.executor:471] Saving best historical training weights: t1_dp0.25/cifar.best.train.w
Validating, loss=1.268: 100%|████████| 6400/6400 [00:00<00:00, 9413.33samples/s]
[INFO 2017-03-02 22:43:01,026 kur.model.executor:197] Validation loss: 1.268
Completed 20 epochs.
[INFO 2017-03-02 22:43:02,257 kur.model.executor:235] Saving most recent weights: t1_dp0.25/cifar.last.w
In [14]:
from IPython.display import Image
In [22]:
Image(width = 500, height=200, retina= True, filename='t1_dp0.25/loss1.png')
Out[22]:
In [24]:
Image(width = 500, height=200, retina= True, filename='t1_dp0.25/loss2.png')
Out[24]:
In [27]:
Image(width = 500, height=500, retina= True, filename = 't1_dp0.25/loss3.png')
Out[27]:
In [4]:
!kur dump dlnd_p2_dropout.yml
{
    "evaluate": {
        "data": [
            {
                "cifar": {
                    "checksum": "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce",
                    "parts": "test",
                    "path": "~/kur",
                    "url": "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
                }
            }
        ],
        "destination": "t1_dp0.25/cifar.results.pkl",
        "provider": {
            "batch_size": 128,
            "num_batches": 1000,
            "randomize": false
        },
        "weights": "t1_dp0.25/cifar.best.valid.w"
    },
    "evaluation": {
        "data": [
            {
                "cifar": {
                    "checksum": "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce",
                    "parts": "test",
                    "path": "~/kur",
                    "url": "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
                }
            }
        ],
        "destination": "t1_dp0.25/cifar.results.pkl",
        "provider": {
            "batch_size": 128,
            "num_batches": 1000,
            "randomize": false
        },
        "weights": "t1_dp0.25/cifar.best.valid.w"
    },
    "loss": [
        {
            "name": "categorical_crossentropy",
            "target": "labels"
        }
    ],
    "model": [
        {
            "input": "images"
        },
        {
            "convolution": {
                "kernels": 20,
                "size": [
                    5,
                    5
                ],
                "strides": [
                    2,
                    2
                ]
            }
        },
        {
            "activation": "relu"
        },
        {
            "pool": {
                "size": [
                    2,
                    2
                ],
                "strides": [
                    2,
                    2
                ],
                "type": "max"
            }
        },
        {
            "flatten": null
        },
        {
            "dense": 15
        },
        {
            "dropout": 0.25
        },
        {
            "dense": 10
        },
        {
            "activation": "softmax",
            "name": "labels"
        }
    ],
    "settings": {
        "backend": {
            "name": "keras"
        },
        "cifar": {
            "checksum": "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce",
            "path": "~/kur",
            "url": "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
        }
    },
    "test": {
        "data": [
            {
                "cifar": {
                    "checksum": "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce",
                    "parts": "test",
                    "path": "~/kur",
                    "url": "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
                }
            }
        ],
        "provider": {
            "batch_size": 128,
            "num_batches": 1000,
            "randomize": false
        },
        "weights": "t1_dp0.25/cifar.best.valid.w"
    },
    "testing": {
        "data": [
            {
                "cifar": {
                    "checksum": "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce",
                    "parts": "test",
                    "path": "~/kur",
                    "url": "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
                }
            }
        ],
        "provider": {
            "batch_size": 128,
            "num_batches": 1000,
            "randomize": false
        },
        "weights": "t1_dp0.25/cifar.best.valid.w"
    },
    "train": {
        "data": [
            {
                "cifar": {
                    "checksum": "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce",
                    "parts": [
                        1
                    ],
                    "path": "~/kur",
                    "url": "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
                }
            }
        ],
        "epochs": 20,
        "hooks": [
            {
                "plot": {
                    "loss_per_batch": "t1_dp0.25/loss1.png",
                    "loss_per_time": "t1_dp0.25/loss2.png",
                    "throughput_per_time": "t1_dp0.25/loss3.png"
                }
            }
        ],
        "log": "t1_dp0.25/cifar-log",
        "optimizer": {
            "learning_rate": 0.001,
            "name": "adam"
        },
        "provider": {
            "batch_size": 128,
            "num_batches": 1000,
            "randomize": true
        },
        "weights": {
            "best": "t1_dp0.25/cifar.best.train.w",
            "initial": "t1_dp0.25/cifar.best.valid.w",
            "last": "t1_dp0.25/cifar.last.w"
        }
    },
    "training": {
        "data": [
            {
                "cifar": {
                    "checksum": "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce",
                    "parts": [
                        1
                    ],
                    "path": "~/kur",
                    "url": "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
                }
            }
        ],
        "epochs": 20,
        "hooks": [
            {
                "plot": {
                    "loss_per_batch": "t1_dp0.25/loss1.png",
                    "loss_per_time": "t1_dp0.25/loss2.png",
                    "throughput_per_time": "t1_dp0.25/loss3.png"
                }
            }
        ],
        "log": "t1_dp0.25/cifar-log",
        "optimizer": {
            "learning_rate": 0.001,
            "name": "adam"
        },
        "provider": {
            "batch_size": 128,
            "num_batches": 1000,
            "randomize": true
        },
        "weights": {
            "best": "t1_dp0.25/cifar.best.train.w",
            "initial": "t1_dp0.25/cifar.best.valid.w",
            "last": "t1_dp0.25/cifar.last.w"
        }
    },
    "validate": {
        "data": [
            {
                "cifar": {
                    "checksum": "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce",
                    "parts": 5,
                    "path": "~/kur",
                    "url": "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
                }
            }
        ],
        "provider": {
            "batch_size": 128,
            "num_batches": 50,
            "randomize": true
        },
        "weights": "t1_dp0.25/cifar.best.valid.w"
    },
    "validation": {
        "data": [
            {
                "cifar": {
                    "checksum": "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce",
                    "parts": 5,
                    "path": "~/kur",
                    "url": "https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz"
                }
            }
        ],
        "provider": {
            "batch_size": 128,
            "num_batches": 50,
            "randomize": true
        },
        "weights": "t1_dp0.25/cifar.best.valid.w"
    }
}
In [ ]: