Решить задачу выбора алгоритма оптимизации на наборе данных MNIST с использованием нейронных сетей простой структуры. Выбор произвести между алгоритмами: SGD, Nesterov Momentum, Adam; по значениям скорости сходимости, значения оптимума и вида траектории. В качестве структурного параметра используется количество нейронов
Выборка состоит из 70000 изображений рукописного написания цифр размера 28×28 пикселей.
Подключение необходимых библиотек
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from keras.datasets import mnist
from keras.utils import np_utils
Using TensorFlow backend.
Загрузка датасета
(X_train, y_train), (X_test, y_test) = mnist.load_data()
data = X_train
labels = y_train
fig = plt.figure(figsize=(8, 6))
for i in range(15):
ax = fig.add_subplot(3, 5, i + 1, xticks=[], yticks=[])
ax.imshow(X_train[i])
Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz 11493376/11490434 [==============================] - 2s 0us/step
Изображение 28×28 пикселей представляется в виде массива размера 784
num_pixels = X_train.shape[1] * X_train.shape[2]
data = data.reshape(X_train.shape[0], num_pixels)
X_train = X_train.reshape(X_train.shape[0], num_pixels).astype('float32')
X_test = X_test.reshape(X_test.shape[0], num_pixels).astype('float32')
data = pd.DataFrame(data=data, # values
index=np.arange(data.shape[0]), # 1st column as index
columns=np.arange(data.shape[1])) # 1st row as the column names
data.head()
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ... | 774 | 775 | 776 | 777 | 778 | 779 | 780 | 781 | 782 | 783 | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
5 rows × 784 columns
Нормализация объектов выборки: из 0-255 в 0-1
X_train = X_train/255
X_test = X_test/255
Классы (10 цифр) представляются в виде бинарных матриц размера 2×2
y_train = np_utils.to_categorical(y_train)
y_test = np_utils.to_categorical(y_test)
num_classes = y_test.shape[1]
sns.countplot(labels)
/usr/local/lib/python3.6/dist-packages/seaborn/categorical.py:1428: FutureWarning: remove_na is deprecated and is a private function. Do not use. stat_data = remove_na(group_data)
<matplotlib.axes._subplots.AxesSubplot at 0x7ff4a961b208>
Выбросов не наблюдается
data.isnull().any().describe()
count 784 unique 1 top False freq 784 dtype: object
В выборке нет пропусков или испорченных данных
#corr_matrix = data.corr()
Не удалось получить наглядную корелляционную матрицу для 784 признаков
Строится нейронная сеть простой стуктуры
Критерии качества, используемые в задаче:
Используется разбиение, предлагаемое функцией load_data() модуля mnist библиотеки keras. А именно, 60000 и 10000 объектов на обучение и контроль соответсвенно.
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.preprocessing.image import ImageDataGenerator
def baseline_model(optimizer='adam', param=num_pixels):
model = Sequential()
model.add(Dense(param, input_dim=num_pixels, kernel_initializer='normal', activation='relu'))
model.add(Dense(num_classes, kernel_initializer='normal', activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
return model
optimizers = ['adam', 'SGD', 'nadam']
accuracy = list()
history = dict()
for optimizer in optimizers:
model = baseline_model(optimizer)
history[optimizer] = model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=200, verbose=1)
scores = model.evaluate(X_test, y_test, verbose=0)
accuracy.append(scores[1])
print("Baseline Error: %.2f%%" % (100-scores[1]*100))
Train on 60000 samples, validate on 10000 samples Epoch 1/10 60000/60000 [==============================] - 8s 127us/step - loss: 0.2807 - acc: 0.9189 - val_loss: 0.1363 - val_acc: 0.9605 Epoch 2/10 60000/60000 [==============================] - 7s 123us/step - loss: 0.1109 - acc: 0.9678 - val_loss: 0.0933 - val_acc: 0.9713 Epoch 3/10 60000/60000 [==============================] - 7s 123us/step - loss: 0.0701 - acc: 0.9799 - val_loss: 0.0811 - val_acc: 0.9747 Epoch 4/10 60000/60000 [==============================] - 7s 123us/step - loss: 0.0494 - acc: 0.9859 - val_loss: 0.0701 - val_acc: 0.9779 Epoch 5/10 60000/60000 [==============================] - 7s 122us/step - loss: 0.0363 - acc: 0.9899 - val_loss: 0.0649 - val_acc: 0.9792 Epoch 6/10 60000/60000 [==============================] - 7s 122us/step - loss: 0.0256 - acc: 0.9933 - val_loss: 0.0637 - val_acc: 0.9796 Epoch 7/10 60000/60000 [==============================] - 7s 122us/step - loss: 0.0185 - acc: 0.9956 - val_loss: 0.0564 - val_acc: 0.9819 Epoch 8/10 60000/60000 [==============================] - 7s 122us/step - loss: 0.0141 - acc: 0.9968 - val_loss: 0.0629 - val_acc: 0.9796 Epoch 9/10 60000/60000 [==============================] - 7s 121us/step - loss: 0.0101 - acc: 0.9980 - val_loss: 0.0608 - val_acc: 0.9818 Epoch 10/10 60000/60000 [==============================] - 7s 121us/step - loss: 0.0078 - acc: 0.9985 - val_loss: 0.0551 - val_acc: 0.9834 Baseline Error: 1.66% Train on 60000 samples, validate on 10000 samples Epoch 1/10 60000/60000 [==============================] - 6s 107us/step - loss: 1.2237 - acc: 0.7211 - val_loss: 0.7076 - val_acc: 0.8509 Epoch 2/10 60000/60000 [==============================] - 6s 102us/step - loss: 0.6059 - acc: 0.8603 - val_loss: 0.4961 - val_acc: 0.8814 Epoch 3/10 60000/60000 [==============================] - 6s 103us/step - loss: 0.4753 - acc: 0.8806 - val_loss: 0.4186 - val_acc: 0.8938 Epoch 4/10 60000/60000 [==============================] - 6s 102us/step - loss: 0.4165 - acc: 0.8913 - val_loss: 0.3767 - val_acc: 0.9009 Epoch 5/10 60000/60000 [==============================] - 6s 101us/step - loss: 0.3814 - acc: 0.8984 - val_loss: 0.3501 - val_acc: 0.9059 Epoch 6/10 60000/60000 [==============================] - 6s 104us/step - loss: 0.3573 - acc: 0.9031 - val_loss: 0.3306 - val_acc: 0.9104 Epoch 7/10 60000/60000 [==============================] - 6s 105us/step - loss: 0.3392 - acc: 0.9069 - val_loss: 0.3157 - val_acc: 0.9141 Epoch 8/10 60000/60000 [==============================] - 6s 103us/step - loss: 0.3246 - acc: 0.9108 - val_loss: 0.3040 - val_acc: 0.9166 Epoch 9/10 60000/60000 [==============================] - 6s 102us/step - loss: 0.3128 - acc: 0.9134 - val_loss: 0.2939 - val_acc: 0.9191 Epoch 10/10 60000/60000 [==============================] - 6s 103us/step - loss: 0.3025 - acc: 0.9163 - val_loss: 0.2849 - val_acc: 0.9215 Baseline Error: 7.85% Train on 60000 samples, validate on 10000 samples Epoch 1/10 60000/60000 [==============================] - 8s 130us/step - loss: 0.2381 - acc: 0.9296 - val_loss: 0.1090 - val_acc: 0.9672 Epoch 2/10 60000/60000 [==============================] - 7s 124us/step - loss: 0.0834 - acc: 0.9748 - val_loss: 0.0771 - val_acc: 0.9770 Epoch 3/10 60000/60000 [==============================] - 7s 121us/step - loss: 0.0511 - acc: 0.9844 - val_loss: 0.0702 - val_acc: 0.9791 Epoch 4/10 60000/60000 [==============================] - 7s 123us/step - loss: 0.0331 - acc: 0.9902 - val_loss: 0.0633 - val_acc: 0.9815 Epoch 5/10 60000/60000 [==============================] - 7s 123us/step - loss: 0.0226 - acc: 0.9931 - val_loss: 0.0570 - val_acc: 0.9822 Epoch 6/10 60000/60000 [==============================] - 7s 123us/step - loss: 0.0150 - acc: 0.9960 - val_loss: 0.0656 - val_acc: 0.9807 Epoch 7/10 60000/60000 [==============================] - 7s 123us/step - loss: 0.0124 - acc: 0.9964 - val_loss: 0.0637 - val_acc: 0.9813 Epoch 8/10 60000/60000 [==============================] - 7s 124us/step - loss: 0.0095 - acc: 0.9975 - val_loss: 0.0672 - val_acc: 0.9808 Epoch 9/10 60000/60000 [==============================] - 7s 123us/step - loss: 0.0096 - acc: 0.9974 - val_loss: 0.0669 - val_acc: 0.9809 Epoch 10/10 60000/60000 [==============================] - 7s 123us/step - loss: 0.0099 - acc: 0.9967 - val_loss: 0.0767 - val_acc: 0.9801 Baseline Error: 1.99%
По значению оптимума (Baseline Error) лучшим является алгоритм ADAM
for optimizer in optimizers:
# summarize history for accuracy
t_train_acc = np.array(history[optimizer].history['acc'])
std_train_acc = t_train_acc.std()
t_test_acc = np.array(history[optimizer].history['val_acc'])
std_test_acc = t_test_acc.std()
plt.plot(t_train_acc)
plt.fill_between(np.arange(t_train_acc.shape[0]), t_train_acc+std_train_acc,
t_train_acc-std_train_acc, facecolor='blue', alpha=0.5)
plt.plot(t_test_acc)
plt.fill_between(np.arange(t_test_acc.shape[0]), t_test_acc+std_test_acc,
t_test_acc-std_test_acc, facecolor='green', alpha=0.5)
plt.title('{} accuracy'.format(optimizer))
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
t_train_loss = np.array(history[optimizer].history['loss'])
std_train_loss = t_train_loss.std()
t_test_loss = np.array(history[optimizer].history['val_loss'])
std_test_loss = t_test_loss.std()
plt.plot(t_train_loss)
plt.fill_between(np.arange(t_train_loss.shape[0]), t_train_loss+std_train_loss,
t_train_loss-std_train_loss, facecolor='blue', alpha=0.5)
plt.plot(t_test_loss)
plt.fill_between(np.arange(t_test_loss.shape[0]), t_test_loss+std_test_loss,
t_test_loss-std_test_loss, facecolor='green', alpha=0.5)
plt.title('{} loss'.format(optimizer))
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
По скорости сходимости также выигрывает ADAM
В качестве структурных параметров выбраны число нейронов во входном слое и размер пачки входящих объектов (batch_size). Рассмотрим 5 значений первого параметра от 10 до 784 и также 5 для второго: от 10 до 300. Для увеличения сходимости уменьшено количество раундов, что негативно влияет на точность
optimizers = ['adam', 'SGD', 'nadam']
params = np.linspace(10, 784, num=5, dtype='Int32')
batch_size_list = np.linspace(10, 300, num=5, dtype='Int32')
accuracy = {optimizer: list() for optimizer in optimizers}
for optimizer in optimizers:
for param in params:
for batch_size in batch_size_list:
model = baseline_model(optimizer, param)
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=4, batch_size=batch_size, verbose=1)
scores = model.evaluate(X_test, y_test, verbose=0)
accuracy[optimizer].append(scores[1])
print("Baseline Error: %.2f%%" % (100-scores[1]*100))
Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 23s 386us/step - loss: 0.4331 - acc: 0.8773 - val_loss: 0.2931 - val_acc: 0.9171 Epoch 2/4 60000/60000 [==============================] - 17s 284us/step - loss: 0.2833 - acc: 0.9178 - val_loss: 0.2640 - val_acc: 0.9252 Epoch 3/4 60000/60000 [==============================] - 17s 284us/step - loss: 0.2579 - acc: 0.9260 - val_loss: 0.2509 - val_acc: 0.9267 Epoch 4/4 60000/60000 [==============================] - 17s 290us/step - loss: 0.2439 - acc: 0.9301 - val_loss: 0.2516 - val_acc: 0.9274 Baseline Error: 7.26% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 9s 154us/step - loss: 0.7801 - acc: 0.7707 - val_loss: 0.3635 - val_acc: 0.8945 Epoch 2/4 60000/60000 [==============================] - 3s 45us/step - loss: 0.3317 - acc: 0.9053 - val_loss: 0.2964 - val_acc: 0.9168 Epoch 3/4 60000/60000 [==============================] - 3s 43us/step - loss: 0.2906 - acc: 0.9177 - val_loss: 0.2716 - val_acc: 0.9243 Epoch 4/4 60000/60000 [==============================] - 3s 45us/step - loss: 0.2723 - acc: 0.9230 - val_loss: 0.2608 - val_acc: 0.9275 Baseline Error: 7.25% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 8s 138us/step - loss: 0.9423 - acc: 0.7576 - val_loss: 0.4112 - val_acc: 0.8921 Epoch 2/4 60000/60000 [==============================] - 2s 29us/step - loss: 0.3684 - acc: 0.8981 - val_loss: 0.3180 - val_acc: 0.9103 Epoch 3/4 60000/60000 [==============================] - 2s 27us/step - loss: 0.3147 - acc: 0.9115 - val_loss: 0.2938 - val_acc: 0.9198 Epoch 4/4 60000/60000 [==============================] - 2s 29us/step - loss: 0.2926 - acc: 0.9169 - val_loss: 0.2793 - val_acc: 0.9212 Baseline Error: 7.88% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 8s 137us/step - loss: 1.1739 - acc: 0.6716 - val_loss: 0.5077 - val_acc: 0.8716 Epoch 2/4 60000/60000 [==============================] - 1s 24us/step - loss: 0.4186 - acc: 0.8869 - val_loss: 0.3495 - val_acc: 0.9040 Epoch 3/4 60000/60000 [==============================] - 1s 21us/step - loss: 0.3372 - acc: 0.9043 - val_loss: 0.3083 - val_acc: 0.9122 Epoch 4/4 60000/60000 [==============================] - 1s 22us/step - loss: 0.3066 - acc: 0.9134 - val_loss: 0.2893 - val_acc: 0.9210 Baseline Error: 7.90% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 8s 136us/step - loss: 1.3171 - acc: 0.6323 - val_loss: 0.5691 - val_acc: 0.8595 Epoch 2/4 60000/60000 [==============================] - 1s 19us/step - loss: 0.4606 - acc: 0.8778 - val_loss: 0.3735 - val_acc: 0.9005 Epoch 3/4 60000/60000 [==============================] - 1s 20us/step - loss: 0.3587 - acc: 0.8994 - val_loss: 0.3233 - val_acc: 0.9112 Epoch 4/4 60000/60000 [==============================] - 1s 20us/step - loss: 0.3214 - acc: 0.9087 - val_loss: 0.2985 - val_acc: 0.9162 Baseline Error: 8.38% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 49s 816us/step - loss: 0.2068 - acc: 0.9389 - val_loss: 0.1101 - val_acc: 0.9659 Epoch 2/4 60000/60000 [==============================] - 42s 703us/step - loss: 0.0862 - acc: 0.9734 - val_loss: 0.0797 - val_acc: 0.9742 Epoch 3/4 60000/60000 [==============================] - 42s 697us/step - loss: 0.0590 - acc: 0.9818 - val_loss: 0.0711 - val_acc: 0.9791 Epoch 4/4 60000/60000 [==============================] - 47s 782us/step - loss: 0.0447 - acc: 0.9851 - val_loss: 0.0685 - val_acc: 0.9785 Baseline Error: 2.15% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 14s 236us/step - loss: 0.3243 - acc: 0.9093 - val_loss: 0.1665 - val_acc: 0.9509 Epoch 2/4 60000/60000 [==============================] - 7s 108us/step - loss: 0.1395 - acc: 0.9601 - val_loss: 0.1169 - val_acc: 0.9638 Epoch 3/4 60000/60000 [==============================] - 7s 110us/step - loss: 0.0957 - acc: 0.9719 - val_loss: 0.0983 - val_acc: 0.9693 Epoch 4/4 60000/60000 [==============================] - 7s 112us/step - loss: 0.0706 - acc: 0.9792 - val_loss: 0.0786 - val_acc: 0.9757 Baseline Error: 2.43% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 12s 204us/step - loss: 0.3785 - acc: 0.8989 - val_loss: 0.1931 - val_acc: 0.9439 Epoch 2/4 60000/60000 [==============================] - 4s 68us/step - loss: 0.1630 - acc: 0.9529 - val_loss: 0.1403 - val_acc: 0.9588 Epoch 3/4 60000/60000 [==============================] - 4s 71us/step - loss: 0.1129 - acc: 0.9674 - val_loss: 0.1090 - val_acc: 0.9673 Epoch 4/4 60000/60000 [==============================] - 4s 72us/step - loss: 0.0863 - acc: 0.9751 - val_loss: 0.0911 - val_acc: 0.9720 Baseline Error: 2.80% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 11s 187us/step - loss: 0.4326 - acc: 0.8868 - val_loss: 0.2241 - val_acc: 0.9355 Epoch 2/4 60000/60000 [==============================] - 3s 55us/step - loss: 0.1862 - acc: 0.9475 - val_loss: 0.1531 - val_acc: 0.9560 Epoch 3/4 60000/60000 [==============================] - 3s 54us/step - loss: 0.1332 - acc: 0.9619 - val_loss: 0.1218 - val_acc: 0.9654 Epoch 4/4 60000/60000 [==============================] - 3s 53us/step - loss: 0.1026 - acc: 0.9707 - val_loss: 0.0999 - val_acc: 0.9694 Baseline Error: 3.06% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 11s 177us/step - loss: 0.4911 - acc: 0.8735 - val_loss: 0.2450 - val_acc: 0.9295 Epoch 2/4 60000/60000 [==============================] - 3s 45us/step - loss: 0.2110 - acc: 0.9403 - val_loss: 0.1749 - val_acc: 0.9498 Epoch 3/4 60000/60000 [==============================] - 3s 47us/step - loss: 0.1555 - acc: 0.9560 - val_loss: 0.1387 - val_acc: 0.9591 Epoch 4/4 60000/60000 [==============================] - 3s 45us/step - loss: 0.1224 - acc: 0.9652 - val_loss: 0.1171 - val_acc: 0.9643 Baseline Error: 3.57% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 67s 1ms/step - loss: 0.1875 - acc: 0.9439 - val_loss: 0.0988 - val_acc: 0.9680 Epoch 2/4 60000/60000 [==============================] - 59s 981us/step - loss: 0.0787 - acc: 0.9757 - val_loss: 0.0832 - val_acc: 0.9744 Epoch 3/4 60000/60000 [==============================] - 60s 1ms/step - loss: 0.0551 - acc: 0.9825 - val_loss: 0.0670 - val_acc: 0.9790 Epoch 4/4 60000/60000 [==============================] - 61s 1ms/step - loss: 0.0411 - acc: 0.9862 - val_loss: 0.0743 - val_acc: 0.9777 Baseline Error: 2.23% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 17s 282us/step - loss: 0.2667 - acc: 0.9237 - val_loss: 0.1323 - val_acc: 0.9592 Epoch 2/4 60000/60000 [==============================] - 9s 151us/step - loss: 0.1055 - acc: 0.9694 - val_loss: 0.0907 - val_acc: 0.9718 Epoch 3/4 60000/60000 [==============================] - 9s 150us/step - loss: 0.0690 - acc: 0.9796 - val_loss: 0.0752 - val_acc: 0.9771 Epoch 4/4 60000/60000 [==============================] - 9s 150us/step - loss: 0.0479 - acc: 0.9857 - val_loss: 0.0780 - val_acc: 0.9764 Baseline Error: 2.36% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 14s 225us/step - loss: 0.3111 - acc: 0.9146 - val_loss: 0.1514 - val_acc: 0.9557 Epoch 2/4 60000/60000 [==============================] - 6s 96us/step - loss: 0.1281 - acc: 0.9628 - val_loss: 0.1081 - val_acc: 0.9681 Epoch 3/4 60000/60000 [==============================] - 6s 96us/step - loss: 0.0852 - acc: 0.9754 - val_loss: 0.0863 - val_acc: 0.9739 Epoch 4/4 60000/60000 [==============================] - 6s 97us/step - loss: 0.0628 - acc: 0.9820 - val_loss: 0.0761 - val_acc: 0.9764 Baseline Error: 2.36% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 13s 210us/step - loss: 0.3505 - acc: 0.9048 - val_loss: 0.1795 - val_acc: 0.9491 Epoch 2/4 60000/60000 [==============================] - 5s 78us/step - loss: 0.1473 - acc: 0.9580 - val_loss: 0.1236 - val_acc: 0.9636 Epoch 3/4 60000/60000 [==============================] - 5s 79us/step - loss: 0.0995 - acc: 0.9720 - val_loss: 0.1008 - val_acc: 0.9699 Epoch 4/4 60000/60000 [==============================] - 5s 78us/step - loss: 0.0744 - acc: 0.9788 - val_loss: 0.0874 - val_acc: 0.9736 Baseline Error: 2.64% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 12s 203us/step - loss: 0.3998 - acc: 0.8903 - val_loss: 0.1914 - val_acc: 0.9457 Epoch 2/4 60000/60000 [==============================] - 4s 69us/step - loss: 0.1638 - acc: 0.9533 - val_loss: 0.1298 - val_acc: 0.9623 Epoch 3/4 60000/60000 [==============================] - 4s 69us/step - loss: 0.1119 - acc: 0.9679 - val_loss: 0.1010 - val_acc: 0.9699 Epoch 4/4 60000/60000 [==============================] - 4s 72us/step - loss: 0.0839 - acc: 0.9765 - val_loss: 0.0921 - val_acc: 0.9721 Baseline Error: 2.79% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 80s 1ms/step - loss: 0.1791 - acc: 0.9457 - val_loss: 0.1100 - val_acc: 0.9649 Epoch 2/4 60000/60000 [==============================] - 67s 1ms/step - loss: 0.0795 - acc: 0.9758 - val_loss: 0.0785 - val_acc: 0.9766 Epoch 3/4 60000/60000 [==============================] - 75s 1ms/step - loss: 0.0572 - acc: 0.9823 - val_loss: 0.0914 - val_acc: 0.9771 Epoch 4/4 60000/60000 [==============================] - 74s 1ms/step - loss: 0.0409 - acc: 0.9866 - val_loss: 0.0881 - val_acc: 0.9784 Baseline Error: 2.16% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 21s 343us/step - loss: 0.2368 - acc: 0.9309 - val_loss: 0.1202 - val_acc: 0.9632 Epoch 2/4 60000/60000 [==============================] - 12s 202us/step - loss: 0.0919 - acc: 0.9730 - val_loss: 0.0842 - val_acc: 0.9736 Epoch 3/4 60000/60000 [==============================] - 12s 204us/step - loss: 0.0582 - acc: 0.9827 - val_loss: 0.0697 - val_acc: 0.9785 Epoch 4/4 60000/60000 [==============================] - 11s 191us/step - loss: 0.0396 - acc: 0.9884 - val_loss: 0.0633 - val_acc: 0.9797 Baseline Error: 2.03% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 16s 264us/step - loss: 0.2796 - acc: 0.9221 - val_loss: 0.1381 - val_acc: 0.9607 Epoch 2/4 60000/60000 [==============================] - 7s 124us/step - loss: 0.1142 - acc: 0.9669 - val_loss: 0.0976 - val_acc: 0.9695 Epoch 3/4 60000/60000 [==============================] - 8s 128us/step - loss: 0.0730 - acc: 0.9785 - val_loss: 0.0810 - val_acc: 0.9752 Epoch 4/4 60000/60000 [==============================] - 8s 128us/step - loss: 0.0517 - acc: 0.9848 - val_loss: 0.0722 - val_acc: 0.9762 Baseline Error: 2.38% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 14s 229us/step - loss: 0.3068 - acc: 0.9148 - val_loss: 0.1511 - val_acc: 0.9558 Epoch 2/4 60000/60000 [==============================] - 6s 98us/step - loss: 0.1246 - acc: 0.9640 - val_loss: 0.1077 - val_acc: 0.9675 Epoch 3/4 60000/60000 [==============================] - 6s 102us/step - loss: 0.0824 - acc: 0.9765 - val_loss: 0.0807 - val_acc: 0.9747 Epoch 4/4 60000/60000 [==============================] - 6s 104us/step - loss: 0.0594 - acc: 0.9830 - val_loss: 0.0706 - val_acc: 0.9776 Baseline Error: 2.24% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 13s 225us/step - loss: 0.3437 - acc: 0.9048 - val_loss: 0.1792 - val_acc: 0.9487 Epoch 2/4 60000/60000 [==============================] - 5s 90us/step - loss: 0.1399 - acc: 0.9605 - val_loss: 0.1157 - val_acc: 0.9651 Epoch 3/4 60000/60000 [==============================] - 5s 89us/step - loss: 0.0928 - acc: 0.9738 - val_loss: 0.0869 - val_acc: 0.9732 Epoch 4/4 60000/60000 [==============================] - 5s 86us/step - loss: 0.0680 - acc: 0.9810 - val_loss: 0.0785 - val_acc: 0.9749 Baseline Error: 2.51% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 94s 2ms/step - loss: 0.1783 - acc: 0.9469 - val_loss: 0.0961 - val_acc: 0.9682 Epoch 2/4 60000/60000 [==============================] - 81s 1ms/step - loss: 0.0784 - acc: 0.9758 - val_loss: 0.0896 - val_acc: 0.9722 Epoch 3/4 60000/60000 [==============================] - 81s 1ms/step - loss: 0.0539 - acc: 0.9831 - val_loss: 0.0904 - val_acc: 0.9755 Epoch 4/4 60000/60000 [==============================] - 80s 1ms/step - loss: 0.0425 - acc: 0.9868 - val_loss: 0.0996 - val_acc: 0.9755 Baseline Error: 2.45% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 21s 347us/step - loss: 0.2222 - acc: 0.9362 - val_loss: 0.1035 - val_acc: 0.9687 Epoch 2/4 60000/60000 [==============================] - 13s 216us/step - loss: 0.0851 - acc: 0.9743 - val_loss: 0.0805 - val_acc: 0.9760 Epoch 3/4 60000/60000 [==============================] - 13s 213us/step - loss: 0.0532 - acc: 0.9833 - val_loss: 0.0681 - val_acc: 0.9788 Epoch 4/4 60000/60000 [==============================] - 13s 215us/step - loss: 0.0364 - acc: 0.9891 - val_loss: 0.0736 - val_acc: 0.9766 Baseline Error: 2.34% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 16s 274us/step - loss: 0.2594 - acc: 0.9266 - val_loss: 0.1370 - val_acc: 0.9588 Epoch 2/4 60000/60000 [==============================] - 9s 142us/step - loss: 0.1022 - acc: 0.9703 - val_loss: 0.0942 - val_acc: 0.9718 Epoch 3/4 60000/60000 [==============================] - 9s 142us/step - loss: 0.0651 - acc: 0.9809 - val_loss: 0.0743 - val_acc: 0.9766 Epoch 4/4 60000/60000 [==============================] - 9s 143us/step - loss: 0.0450 - acc: 0.9870 - val_loss: 0.0737 - val_acc: 0.9766 Baseline Error: 2.34% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 15s 251us/step - loss: 0.2875 - acc: 0.9178 - val_loss: 0.1454 - val_acc: 0.9578 Epoch 2/4 60000/60000 [==============================] - 7s 117us/step - loss: 0.1160 - acc: 0.9660 - val_loss: 0.1019 - val_acc: 0.9699 Epoch 3/4 60000/60000 [==============================] - 7s 121us/step - loss: 0.0747 - acc: 0.9786 - val_loss: 0.0871 - val_acc: 0.9743 Epoch 4/4 60000/60000 [==============================] - 8s 128us/step - loss: 0.0543 - acc: 0.9842 - val_loss: 0.0672 - val_acc: 0.9787 Baseline Error: 2.13% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 14s 237us/step - loss: 0.3102 - acc: 0.9140 - val_loss: 0.1537 - val_acc: 0.9556 Epoch 2/4 60000/60000 [==============================] - 6s 101us/step - loss: 0.1269 - acc: 0.9635 - val_loss: 0.1056 - val_acc: 0.9684 Epoch 3/4 60000/60000 [==============================] - 6s 100us/step - loss: 0.0831 - acc: 0.9762 - val_loss: 0.0923 - val_acc: 0.9721 Epoch 4/4 60000/60000 [==============================] - 6s 101us/step - loss: 0.0609 - acc: 0.9822 - val_loss: 0.0762 - val_acc: 0.9765 Baseline Error: 2.35% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 27s 447us/step - loss: 0.7449 - acc: 0.7762 - val_loss: 0.3359 - val_acc: 0.9052 Epoch 2/4 60000/60000 [==============================] - 19s 319us/step - loss: 0.3321 - acc: 0.9058 - val_loss: 0.2944 - val_acc: 0.9168 Epoch 3/4 60000/60000 [==============================] - 19s 311us/step - loss: 0.2992 - acc: 0.9151 - val_loss: 0.2800 - val_acc: 0.9199 Epoch 4/4 60000/60000 [==============================] - 19s 311us/step - loss: 0.2802 - acc: 0.9210 - val_loss: 0.2654 - val_acc: 0.9257 Baseline Error: 7.43% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 11s 180us/step - loss: 2.0483 - acc: 0.3589 - val_loss: 1.4207 - val_acc: 0.6214 Epoch 2/4 60000/60000 [==============================] - 3s 46us/step - loss: 0.9538 - acc: 0.7568 - val_loss: 0.6679 - val_acc: 0.8211 Epoch 3/4 60000/60000 [==============================] - 3s 46us/step - loss: 0.5902 - acc: 0.8390 - val_loss: 0.4993 - val_acc: 0.8618 Epoch 4/4 60000/60000 [==============================] - 3s 46us/step - loss: 0.4820 - acc: 0.8666 - val_loss: 0.4306 - val_acc: 0.8789 Baseline Error: 12.11% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 10s 160us/step - loss: 2.2133 - acc: 0.3070 - val_loss: 2.0148 - val_acc: 0.4531 Epoch 2/4 60000/60000 [==============================] - 2s 29us/step - loss: 1.6063 - acc: 0.5901 - val_loss: 1.1771 - val_acc: 0.7147 Epoch 3/4 60000/60000 [==============================] - 2s 28us/step - loss: 0.9604 - acc: 0.7667 - val_loss: 0.7671 - val_acc: 0.8056 Epoch 4/4 60000/60000 [==============================] - 2s 28us/step - loss: 0.6945 - acc: 0.8205 - val_loss: 0.6000 - val_acc: 0.8418 Baseline Error: 15.82% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 9s 153us/step - loss: 2.2483 - acc: 0.2104 - val_loss: 2.1607 - val_acc: 0.3068 Epoch 2/4 60000/60000 [==============================] - 1s 21us/step - loss: 1.9651 - acc: 0.4548 - val_loss: 1.7020 - val_acc: 0.5883 Epoch 3/4 60000/60000 [==============================] - 1s 21us/step - loss: 1.4368 - acc: 0.6455 - val_loss: 1.1791 - val_acc: 0.6990 Epoch 4/4 60000/60000 [==============================] - 1s 21us/step - loss: 1.0430 - acc: 0.7207 - val_loss: 0.9024 - val_acc: 0.7502 Baseline Error: 24.98% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 9s 151us/step - loss: 2.2762 - acc: 0.2483 - val_loss: 2.2399 - val_acc: 0.3064 Epoch 2/4 60000/60000 [==============================] - 1s 18us/step - loss: 2.1581 - acc: 0.3300 - val_loss: 2.0351 - val_acc: 0.3888 Epoch 3/4 60000/60000 [==============================] - 1s 18us/step - loss: 1.8489 - acc: 0.4911 - val_loss: 1.6163 - val_acc: 0.5949 Epoch 4/4 60000/60000 [==============================] - 1s 18us/step - loss: 1.4073 - acc: 0.6479 - val_loss: 1.1947 - val_acc: 0.6944 Baseline Error: 30.56% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 37s 610us/step - loss: 0.4730 - acc: 0.8741 - val_loss: 0.2759 - val_acc: 0.9201 Epoch 2/4 60000/60000 [==============================] - 28s 475us/step - loss: 0.2501 - acc: 0.9283 - val_loss: 0.2158 - val_acc: 0.9377 Epoch 3/4 60000/60000 [==============================] - 29s 482us/step - loss: 0.1986 - acc: 0.9436 - val_loss: 0.1800 - val_acc: 0.9492 Epoch 4/4 60000/60000 [==============================] - 29s 487us/step - loss: 0.1642 - acc: 0.9538 - val_loss: 0.1520 - val_acc: 0.9549 Baseline Error: 4.51% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 12s 207us/step - loss: 1.2018 - acc: 0.7278 - val_loss: 0.6155 - val_acc: 0.8580 Epoch 2/4 60000/60000 [==============================] - 4s 72us/step - loss: 0.5195 - acc: 0.8684 - val_loss: 0.4282 - val_acc: 0.8887 Epoch 3/4 60000/60000 [==============================] - 4s 71us/step - loss: 0.4112 - acc: 0.8885 - val_loss: 0.3673 - val_acc: 0.9005 Epoch 4/4 60000/60000 [==============================] - 4s 71us/step - loss: 0.3651 - acc: 0.8987 - val_loss: 0.3342 - val_acc: 0.9075 Baseline Error: 9.25% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 11s 181us/step - loss: 1.5699 - acc: 0.6484 - val_loss: 0.9508 - val_acc: 0.8095 Epoch 2/4 60000/60000 [==============================] - 3s 48us/step - loss: 0.7515 - acc: 0.8337 - val_loss: 0.5880 - val_acc: 0.8598 Epoch 3/4 60000/60000 [==============================] - 3s 47us/step - loss: 0.5430 - acc: 0.8654 - val_loss: 0.4703 - val_acc: 0.8803 Epoch 4/4 60000/60000 [==============================] - 3s 47us/step - loss: 0.4605 - acc: 0.8799 - val_loss: 0.4142 - val_acc: 0.8894 Baseline Error: 11.06% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 10s 173us/step - loss: 1.8062 - acc: 0.5692 - val_loss: 1.2749 - val_acc: 0.7718 Epoch 2/4 60000/60000 [==============================] - 2s 39us/step - loss: 0.9885 - acc: 0.8012 - val_loss: 0.7568 - val_acc: 0.8355 Epoch 3/4 60000/60000 [==============================] - 2s 39us/step - loss: 0.6785 - acc: 0.8436 - val_loss: 0.5797 - val_acc: 0.8627 Epoch 4/4 60000/60000 [==============================] - 2s 41us/step - loss: 0.5553 - acc: 0.8628 - val_loss: 0.4951 - val_acc: 0.8786 Baseline Error: 12.14% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 10s 174us/step - loss: 1.8976 - acc: 0.5493 - val_loss: 1.4810 - val_acc: 0.7203 Epoch 2/4 60000/60000 [==============================] - 2s 35us/step - loss: 1.1966 - acc: 0.7625 - val_loss: 0.9387 - val_acc: 0.8108 Epoch 3/4 60000/60000 [==============================] - 2s 35us/step - loss: 0.8285 - acc: 0.8200 - val_loss: 0.7006 - val_acc: 0.8455 Epoch 4/4 60000/60000 [==============================] - 2s 35us/step - loss: 0.6594 - acc: 0.8460 - val_loss: 0.5819 - val_acc: 0.8645 Baseline Error: 13.55% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 42s 700us/step - loss: 0.4361 - acc: 0.8842 - val_loss: 0.2604 - val_acc: 0.9275 Epoch 2/4 60000/60000 [==============================] - 34s 564us/step - loss: 0.2389 - acc: 0.9332 - val_loss: 0.2084 - val_acc: 0.9399 Epoch 3/4 60000/60000 [==============================] - 35s 583us/step - loss: 0.1870 - acc: 0.9465 - val_loss: 0.1680 - val_acc: 0.9501 Epoch 4/4 60000/60000 [==============================] - 34s 560us/step - loss: 0.1538 - acc: 0.9572 - val_loss: 0.1440 - val_acc: 0.9576 Baseline Error: 4.24% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 14s 233us/step - loss: 0.9956 - acc: 0.7832 - val_loss: 0.5180 - val_acc: 0.8754 Epoch 2/4 60000/60000 [==============================] - 6s 95us/step - loss: 0.4596 - acc: 0.8825 - val_loss: 0.3858 - val_acc: 0.8982 Epoch 3/4 60000/60000 [==============================] - 6s 94us/step - loss: 0.3786 - acc: 0.8978 - val_loss: 0.3391 - val_acc: 0.9090 Epoch 4/4 60000/60000 [==============================] - 6s 94us/step - loss: 0.3407 - acc: 0.9061 - val_loss: 0.3117 - val_acc: 0.9150 Baseline Error: 8.50% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 12s 206us/step - loss: 1.3704 - acc: 0.6865 - val_loss: 0.7692 - val_acc: 0.8376 Epoch 2/4 60000/60000 [==============================] - 4s 64us/step - loss: 0.6388 - acc: 0.8530 - val_loss: 0.5129 - val_acc: 0.8788 Epoch 3/4 60000/60000 [==============================] - 4s 67us/step - loss: 0.4867 - acc: 0.8764 - val_loss: 0.4276 - val_acc: 0.8903 Epoch 4/4 60000/60000 [==============================] - 4s 69us/step - loss: 0.4232 - acc: 0.8880 - val_loss: 0.3827 - val_acc: 0.8991 Baseline Error: 10.09% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 12s 201us/step - loss: 1.5895 - acc: 0.6203 - val_loss: 1.0070 - val_acc: 0.8115 Epoch 2/4 60000/60000 [==============================] - 4s 60us/step - loss: 0.8087 - acc: 0.8293 - val_loss: 0.6373 - val_acc: 0.8570 Epoch 3/4 60000/60000 [==============================] - 4s 61us/step - loss: 0.5886 - acc: 0.8606 - val_loss: 0.5098 - val_acc: 0.8757 Epoch 4/4 60000/60000 [==============================] - 4s 69us/step - loss: 0.4965 - acc: 0.8755 - val_loss: 0.4444 - val_acc: 0.8890 Baseline Error: 11.10% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 12s 194us/step - loss: 1.6596 - acc: 0.6183 - val_loss: 1.1494 - val_acc: 0.7901 Epoch 2/4 60000/60000 [==============================] - 3s 54us/step - loss: 0.9375 - acc: 0.8097 - val_loss: 0.7378 - val_acc: 0.8410 Epoch 3/4 60000/60000 [==============================] - 3s 53us/step - loss: 0.6768 - acc: 0.8472 - val_loss: 0.5804 - val_acc: 0.8625 Epoch 4/4 60000/60000 [==============================] - 3s 52us/step - loss: 0.5619 - acc: 0.8647 - val_loss: 0.5007 - val_acc: 0.8739 Baseline Error: 12.61% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 49s 824us/step - loss: 0.4041 - acc: 0.8911 - val_loss: 0.2546 - val_acc: 0.9261 Epoch 2/4 60000/60000 [==============================] - 39s 651us/step - loss: 0.2243 - acc: 0.9367 - val_loss: 0.1867 - val_acc: 0.9473 Epoch 3/4 60000/60000 [==============================] - 40s 662us/step - loss: 0.1740 - acc: 0.9509 - val_loss: 0.1512 - val_acc: 0.9564 Epoch 4/4 60000/60000 [==============================] - 38s 632us/step - loss: 0.1427 - acc: 0.9603 - val_loss: 0.1352 - val_acc: 0.9618 Baseline Error: 3.82% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 15s 256us/step - loss: 0.9168 - acc: 0.7867 - val_loss: 0.4854 - val_acc: 0.8772 Epoch 2/4 60000/60000 [==============================] - 7s 117us/step - loss: 0.4394 - acc: 0.8861 - val_loss: 0.3687 - val_acc: 0.9014 Epoch 3/4 60000/60000 [==============================] - 7s 117us/step - loss: 0.3650 - acc: 0.8999 - val_loss: 0.3246 - val_acc: 0.9124 Epoch 4/4 60000/60000 [==============================] - 7s 119us/step - loss: 0.3286 - acc: 0.9090 - val_loss: 0.2989 - val_acc: 0.9185 Baseline Error: 8.15% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 14s 227us/step - loss: 1.1911 - acc: 0.7372 - val_loss: 0.6688 - val_acc: 0.8570 Epoch 2/4 60000/60000 [==============================] - 5s 83us/step - loss: 0.5728 - acc: 0.8644 - val_loss: 0.4712 - val_acc: 0.8878 Epoch 3/4 60000/60000 [==============================] - 5s 82us/step - loss: 0.4528 - acc: 0.8833 - val_loss: 0.4012 - val_acc: 0.8991 Epoch 4/4 60000/60000 [==============================] - 5s 83us/step - loss: 0.3997 - acc: 0.8947 - val_loss: 0.3619 - val_acc: 0.9056 Baseline Error: 9.44% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 13s 215us/step - loss: 1.3830 - acc: 0.6863 - val_loss: 0.8429 - val_acc: 0.8345 Epoch 2/4 60000/60000 [==============================] - 4s 71us/step - loss: 0.7013 - acc: 0.8483 - val_loss: 0.5659 - val_acc: 0.8729 Epoch 3/4 60000/60000 [==============================] - 4s 72us/step - loss: 0.5331 - acc: 0.8718 - val_loss: 0.4647 - val_acc: 0.8901 Epoch 4/4 60000/60000 [==============================] - 4s 71us/step - loss: 0.4590 - acc: 0.8834 - val_loss: 0.4128 - val_acc: 0.8984 Baseline Error: 10.16% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 13s 209us/step - loss: 1.5265 - acc: 0.6353 - val_loss: 0.9959 - val_acc: 0.8093 Epoch 2/4 60000/60000 [==============================] - 4s 67us/step - loss: 0.8244 - acc: 0.8248 - val_loss: 0.6609 - val_acc: 0.8580 Epoch 3/4 60000/60000 [==============================] - 4s 70us/step - loss: 0.6152 - acc: 0.8560 - val_loss: 0.5338 - val_acc: 0.8775 Epoch 4/4 60000/60000 [==============================] - 4s 73us/step - loss: 0.5214 - acc: 0.8708 - val_loss: 0.4673 - val_acc: 0.8858 Baseline Error: 11.42% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 51s 849us/step - loss: 0.3826 - acc: 0.8969 - val_loss: 0.2421 - val_acc: 0.9329 Epoch 2/4 60000/60000 [==============================] - 42s 695us/step - loss: 0.2148 - acc: 0.9396 - val_loss: 0.1840 - val_acc: 0.9463 Epoch 3/4 60000/60000 [==============================] - 41s 683us/step - loss: 0.1660 - acc: 0.9534 - val_loss: 0.1475 - val_acc: 0.9575 Epoch 4/4 60000/60000 [==============================] - 41s 688us/step - loss: 0.1353 - acc: 0.9629 - val_loss: 0.1271 - val_acc: 0.9624 Baseline Error: 3.76% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 16s 275us/step - loss: 0.8185 - acc: 0.8140 - val_loss: 0.4506 - val_acc: 0.8859 Epoch 2/4 60000/60000 [==============================] - 8s 130us/step - loss: 0.4145 - acc: 0.8920 - val_loss: 0.3525 - val_acc: 0.9057 Epoch 3/4 60000/60000 [==============================] - 8s 134us/step - loss: 0.3493 - acc: 0.9050 - val_loss: 0.3129 - val_acc: 0.9161 Epoch 4/4 60000/60000 [==============================] - 8s 133us/step - loss: 0.3159 - acc: 0.9130 - val_loss: 0.2889 - val_acc: 0.9202 Baseline Error: 7.98% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 15s 243us/step - loss: 1.1257 - acc: 0.7412 - val_loss: 0.6232 - val_acc: 0.8633 Epoch 2/4 60000/60000 [==============================] - 6s 97us/step - loss: 0.5397 - acc: 0.8712 - val_loss: 0.4467 - val_acc: 0.8886 Epoch 3/4 60000/60000 [==============================] - 6s 97us/step - loss: 0.4311 - acc: 0.8893 - val_loss: 0.3831 - val_acc: 0.9012 Epoch 4/4 60000/60000 [==============================] - 6s 98us/step - loss: 0.3813 - acc: 0.8992 - val_loss: 0.3472 - val_acc: 0.9082 Baseline Error: 9.18% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 14s 228us/step - loss: 1.3008 - acc: 0.7005 - val_loss: 0.7672 - val_acc: 0.8409 Epoch 2/4 60000/60000 [==============================] - 5s 85us/step - loss: 0.6495 - acc: 0.8529 - val_loss: 0.5308 - val_acc: 0.8718 Epoch 3/4 60000/60000 [==============================] - 5s 86us/step - loss: 0.5039 - acc: 0.8752 - val_loss: 0.4438 - val_acc: 0.8877 Epoch 4/4 60000/60000 [==============================] - 5s 85us/step - loss: 0.4385 - acc: 0.8867 - val_loss: 0.3969 - val_acc: 0.8968 Baseline Error: 10.32% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 13s 223us/step - loss: 1.4149 - acc: 0.6758 - val_loss: 0.8792 - val_acc: 0.8368 Epoch 2/4 60000/60000 [==============================] - 5s 79us/step - loss: 0.7384 - acc: 0.8443 - val_loss: 0.5992 - val_acc: 0.8644 Epoch 3/4 60000/60000 [==============================] - 5s 81us/step - loss: 0.5654 - acc: 0.8663 - val_loss: 0.4933 - val_acc: 0.8809 Epoch 4/4 60000/60000 [==============================] - 5s 86us/step - loss: 0.4871 - acc: 0.8781 - val_loss: 0.4376 - val_acc: 0.8893 Baseline Error: 11.07% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 29s 489us/step - loss: 0.3736 - acc: 0.8925 - val_loss: 0.2648 - val_acc: 0.9240 Epoch 2/4 60000/60000 [==============================] - 21s 347us/step - loss: 0.2644 - acc: 0.9236 - val_loss: 0.2666 - val_acc: 0.9235 Epoch 3/4 60000/60000 [==============================] - 21s 346us/step - loss: 0.2421 - acc: 0.9314 - val_loss: 0.2299 - val_acc: 0.9340 Epoch 4/4 60000/60000 [==============================] - 20s 342us/step - loss: 0.2279 - acc: 0.9347 - val_loss: 0.2341 - val_acc: 0.9321 Baseline Error: 6.79% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 12s 196us/step - loss: 0.5524 - acc: 0.8484 - val_loss: 0.3139 - val_acc: 0.9099 Epoch 2/4 60000/60000 [==============================] - 3s 50us/step - loss: 0.2944 - acc: 0.9163 - val_loss: 0.2770 - val_acc: 0.9198 Epoch 3/4 60000/60000 [==============================] - 3s 49us/step - loss: 0.2629 - acc: 0.9251 - val_loss: 0.2533 - val_acc: 0.9289 Epoch 4/4 60000/60000 [==============================] - 3s 50us/step - loss: 0.2442 - acc: 0.9308 - val_loss: 0.2399 - val_acc: 0.9320 Baseline Error: 6.80% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 11s 177us/step - loss: 0.6573 - acc: 0.8281 - val_loss: 0.3469 - val_acc: 0.9002 Epoch 2/4 60000/60000 [==============================] - 2s 30us/step - loss: 0.3137 - acc: 0.9113 - val_loss: 0.2901 - val_acc: 0.9167 Epoch 3/4 60000/60000 [==============================] - 2s 30us/step - loss: 0.2853 - acc: 0.9191 - val_loss: 0.2700 - val_acc: 0.9238 Epoch 4/4 60000/60000 [==============================] - 2s 30us/step - loss: 0.2701 - acc: 0.9237 - val_loss: 0.2632 - val_acc: 0.9273 Baseline Error: 7.27% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 10s 171us/step - loss: 0.8036 - acc: 0.7882 - val_loss: 0.3553 - val_acc: 0.9003 Epoch 2/4 60000/60000 [==============================] - 1s 23us/step - loss: 0.3337 - acc: 0.9057 - val_loss: 0.3035 - val_acc: 0.9122 Epoch 3/4 60000/60000 [==============================] - 1s 22us/step - loss: 0.2981 - acc: 0.9157 - val_loss: 0.2802 - val_acc: 0.9225 Epoch 4/4 60000/60000 [==============================] - 1s 22us/step - loss: 0.2799 - acc: 0.9205 - val_loss: 0.2710 - val_acc: 0.9225 Baseline Error: 7.75% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 10s 169us/step - loss: 0.8963 - acc: 0.7717 - val_loss: 0.3851 - val_acc: 0.9000 Epoch 2/4 60000/60000 [==============================] - 1s 19us/step - loss: 0.3433 - acc: 0.9050 - val_loss: 0.3023 - val_acc: 0.9151 Epoch 3/4 60000/60000 [==============================] - 1s 19us/step - loss: 0.2977 - acc: 0.9155 - val_loss: 0.2828 - val_acc: 0.9210 Epoch 4/4 60000/60000 [==============================] - 1s 19us/step - loss: 0.2792 - acc: 0.9209 - val_loss: 0.2726 - val_acc: 0.9240 Baseline Error: 7.60% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 49s 811us/step - loss: 0.2012 - acc: 0.9400 - val_loss: 0.1274 - val_acc: 0.9637 Epoch 2/4 60000/60000 [==============================] - 40s 663us/step - loss: 0.0987 - acc: 0.9700 - val_loss: 0.1090 - val_acc: 0.9656 Epoch 3/4 60000/60000 [==============================] - 40s 663us/step - loss: 0.0759 - acc: 0.9771 - val_loss: 0.1171 - val_acc: 0.9667 Epoch 4/4 60000/60000 [==============================] - 41s 688us/step - loss: 0.0625 - acc: 0.9809 - val_loss: 0.1045 - val_acc: 0.9741 Baseline Error: 2.59% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 15s 248us/step - loss: 0.2596 - acc: 0.9270 - val_loss: 0.1361 - val_acc: 0.9578 Epoch 2/4 60000/60000 [==============================] - 6s 95us/step - loss: 0.1022 - acc: 0.9699 - val_loss: 0.0990 - val_acc: 0.9703 Epoch 3/4 60000/60000 [==============================] - 6s 102us/step - loss: 0.0671 - acc: 0.9794 - val_loss: 0.0826 - val_acc: 0.9743 Epoch 4/4 60000/60000 [==============================] - 6s 94us/step - loss: 0.0487 - acc: 0.9845 - val_loss: 0.0668 - val_acc: 0.9797 Baseline Error: 2.03% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 13s 216us/step - loss: 0.3070 - acc: 0.9154 - val_loss: 0.2444 - val_acc: 0.9265 Epoch 2/4 60000/60000 [==============================] - 4s 61us/step - loss: 0.1198 - acc: 0.9654 - val_loss: 0.2742 - val_acc: 0.9157 Epoch 3/4 60000/60000 [==============================] - 4s 60us/step - loss: 0.0799 - acc: 0.9766 - val_loss: 0.0848 - val_acc: 0.9729 Epoch 4/4 60000/60000 [==============================] - 4s 61us/step - loss: 0.0573 - acc: 0.9830 - val_loss: 0.1068 - val_acc: 0.9641 Baseline Error: 3.59% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 12s 205us/step - loss: 0.3420 - acc: 0.9063 - val_loss: 0.1802 - val_acc: 0.9461 Epoch 2/4 60000/60000 [==============================] - 3s 46us/step - loss: 0.1381 - acc: 0.9593 - val_loss: 0.1167 - val_acc: 0.9653 Epoch 3/4 60000/60000 [==============================] - 3s 46us/step - loss: 0.0923 - acc: 0.9724 - val_loss: 0.0964 - val_acc: 0.9702 Epoch 4/4 60000/60000 [==============================] - 3s 47us/step - loss: 0.0681 - acc: 0.9802 - val_loss: 0.0802 - val_acc: 0.9753 Baseline Error: 2.47% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 12s 198us/step - loss: 0.3808 - acc: 0.8987 - val_loss: 0.1921 - val_acc: 0.9449 Epoch 2/4 60000/60000 [==============================] - 2s 39us/step - loss: 0.1543 - acc: 0.9568 - val_loss: 0.1214 - val_acc: 0.9645 Epoch 3/4 60000/60000 [==============================] - 2s 39us/step - loss: 0.1046 - acc: 0.9698 - val_loss: 0.0978 - val_acc: 0.9704 Epoch 4/4 60000/60000 [==============================] - 2s 39us/step - loss: 0.0774 - acc: 0.9776 - val_loss: 0.0841 - val_acc: 0.9736 Baseline Error: 2.64% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 60s 1ms/step - loss: 0.1972 - acc: 0.9413 - val_loss: 0.1006 - val_acc: 0.9671 Epoch 2/4 60000/60000 [==============================] - 51s 847us/step - loss: 0.1019 - acc: 0.9706 - val_loss: 0.1091 - val_acc: 0.9695 Epoch 3/4 60000/60000 [==============================] - 50s 841us/step - loss: 0.0806 - acc: 0.9777 - val_loss: 0.0998 - val_acc: 0.9740 Epoch 4/4 60000/60000 [==============================] - 52s 859us/step - loss: 0.0679 - acc: 0.9813 - val_loss: 0.1343 - val_acc: 0.9703 Baseline Error: 2.97% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 18s 295us/step - loss: 0.2146 - acc: 0.9368 - val_loss: 0.1127 - val_acc: 0.9651 Epoch 2/4 60000/60000 [==============================] - 8s 127us/step - loss: 0.0816 - acc: 0.9751 - val_loss: 0.0768 - val_acc: 0.9767 Epoch 3/4 60000/60000 [==============================] - 8s 129us/step - loss: 0.0512 - acc: 0.9840 - val_loss: 0.0762 - val_acc: 0.9761 Epoch 4/4 60000/60000 [==============================] - 8s 129us/step - loss: 0.0369 - acc: 0.9883 - val_loss: 0.0723 - val_acc: 0.9770 Baseline Error: 2.30% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 15s 246us/step - loss: 0.2558 - acc: 0.9274 - val_loss: 0.1567 - val_acc: 0.9514 Epoch 2/4 60000/60000 [==============================] - 5s 84us/step - loss: 0.0937 - acc: 0.9717 - val_loss: 0.1704 - val_acc: 0.9461 Epoch 3/4 60000/60000 [==============================] - 5s 85us/step - loss: 0.0599 - acc: 0.9816 - val_loss: 0.0703 - val_acc: 0.9775 Epoch 4/4 60000/60000 [==============================] - 5s 85us/step - loss: 0.0416 - acc: 0.9873 - val_loss: 0.0927 - val_acc: 0.9708 Baseline Error: 2.92% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 14s 229us/step - loss: 0.2871 - acc: 0.9192 - val_loss: 0.1440 - val_acc: 0.9568 Epoch 2/4 60000/60000 [==============================] - 4s 67us/step - loss: 0.1059 - acc: 0.9692 - val_loss: 0.0948 - val_acc: 0.9721 Epoch 3/4 60000/60000 [==============================] - 4s 68us/step - loss: 0.0679 - acc: 0.9798 - val_loss: 0.0812 - val_acc: 0.9739 Epoch 4/4 60000/60000 [==============================] - 4s 71us/step - loss: 0.0466 - acc: 0.9860 - val_loss: 0.0781 - val_acc: 0.9741 Baseline Error: 2.59% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 14s 226us/step - loss: 0.3149 - acc: 0.9122 - val_loss: 0.1422 - val_acc: 0.9579 Epoch 2/4 60000/60000 [==============================] - 4s 62us/step - loss: 0.1161 - acc: 0.9662 - val_loss: 0.0923 - val_acc: 0.9707 Epoch 3/4 60000/60000 [==============================] - 4s 61us/step - loss: 0.0745 - acc: 0.9784 - val_loss: 0.0749 - val_acc: 0.9760 Epoch 4/4 60000/60000 [==============================] - 4s 62us/step - loss: 0.0528 - acc: 0.9845 - val_loss: 0.0697 - val_acc: 0.9786 Baseline Error: 2.14% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 75s 1ms/step - loss: 0.1978 - acc: 0.9417 - val_loss: 0.1218 - val_acc: 0.9649 Epoch 2/4 60000/60000 [==============================] - 66s 1ms/step - loss: 0.1048 - acc: 0.9702 - val_loss: 0.1175 - val_acc: 0.9686 Epoch 3/4 60000/60000 [==============================] - 68s 1ms/step - loss: 0.0829 - acc: 0.9772 - val_loss: 0.1125 - val_acc: 0.9722 Epoch 4/4 60000/60000 [==============================] - 68s 1ms/step - loss: 0.0721 - acc: 0.9813 - val_loss: 0.1184 - val_acc: 0.9752 Baseline Error: 2.48% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 21s 344us/step - loss: 0.1988 - acc: 0.9419 - val_loss: 0.1033 - val_acc: 0.9688 Epoch 2/4 60000/60000 [==============================] - 11s 175us/step - loss: 0.0755 - acc: 0.9764 - val_loss: 0.0765 - val_acc: 0.9745 Epoch 3/4 60000/60000 [==============================] - 11s 175us/step - loss: 0.0473 - acc: 0.9848 - val_loss: 0.0704 - val_acc: 0.9789 Epoch 4/4 60000/60000 [==============================] - 10s 172us/step - loss: 0.0343 - acc: 0.9893 - val_loss: 0.0747 - val_acc: 0.9785 Baseline Error: 2.15% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 17s 279us/step - loss: 0.2283 - acc: 0.9344 - val_loss: 0.1351 - val_acc: 0.9570 Epoch 2/4 60000/60000 [==============================] - 7s 114us/step - loss: 0.0819 - acc: 0.9757 - val_loss: 0.0915 - val_acc: 0.9719 Epoch 3/4 60000/60000 [==============================] - 7s 114us/step - loss: 0.0501 - acc: 0.9851 - val_loss: 0.0795 - val_acc: 0.9751 Epoch 4/4 60000/60000 [==============================] - 7s 114us/step - loss: 0.0346 - acc: 0.9890 - val_loss: 0.0682 - val_acc: 0.9782 Baseline Error: 2.18% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 15s 258us/step - loss: 0.2626 - acc: 0.9245 - val_loss: 0.1599 - val_acc: 0.9513 Epoch 2/4 60000/60000 [==============================] - 5s 90us/step - loss: 0.0920 - acc: 0.9731 - val_loss: 0.1157 - val_acc: 0.9633 Epoch 3/4 60000/60000 [==============================] - 5s 91us/step - loss: 0.0571 - acc: 0.9837 - val_loss: 0.0753 - val_acc: 0.9762 Epoch 4/4 60000/60000 [==============================] - 5s 90us/step - loss: 0.0382 - acc: 0.9884 - val_loss: 0.0659 - val_acc: 0.9784 Baseline Error: 2.16% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 15s 248us/step - loss: 0.2908 - acc: 0.9173 - val_loss: 0.1326 - val_acc: 0.9619 Epoch 2/4 60000/60000 [==============================] - 5s 79us/step - loss: 0.1055 - acc: 0.9691 - val_loss: 0.0955 - val_acc: 0.9719 Epoch 3/4 60000/60000 [==============================] - 5s 80us/step - loss: 0.0663 - acc: 0.9803 - val_loss: 0.0745 - val_acc: 0.9775 Epoch 4/4 60000/60000 [==============================] - 5s 78us/step - loss: 0.0449 - acc: 0.9871 - val_loss: 0.0682 - val_acc: 0.9791 Baseline Error: 2.09% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 93s 2ms/step - loss: 0.2078 - acc: 0.9396 - val_loss: 0.1312 - val_acc: 0.9614 Epoch 2/4 60000/60000 [==============================] - 80s 1ms/step - loss: 0.1117 - acc: 0.9693 - val_loss: 0.1066 - val_acc: 0.9744 Epoch 3/4 60000/60000 [==============================] - 83s 1ms/step - loss: 0.0918 - acc: 0.9763 - val_loss: 0.1412 - val_acc: 0.9683 Epoch 4/4 60000/60000 [==============================] - 81s 1ms/step - loss: 0.0772 - acc: 0.9811 - val_loss: 0.1036 - val_acc: 0.9786 Baseline Error: 2.14% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 23s 377us/step - loss: 0.1911 - acc: 0.9436 - val_loss: 0.1009 - val_acc: 0.9700 Epoch 2/4 60000/60000 [==============================] - 13s 218us/step - loss: 0.0714 - acc: 0.9781 - val_loss: 0.0743 - val_acc: 0.9764 Epoch 3/4 60000/60000 [==============================] - 13s 209us/step - loss: 0.0460 - acc: 0.9856 - val_loss: 0.0714 - val_acc: 0.9790 Epoch 4/4 60000/60000 [==============================] - 13s 211us/step - loss: 0.0320 - acc: 0.9897 - val_loss: 0.0769 - val_acc: 0.9783 Baseline Error: 2.17% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 19s 311us/step - loss: 0.2164 - acc: 0.9369 - val_loss: 0.3326 - val_acc: 0.9018 Epoch 2/4 60000/60000 [==============================] - 9s 152us/step - loss: 0.0780 - acc: 0.9765 - val_loss: 0.0884 - val_acc: 0.9720 Epoch 3/4 60000/60000 [==============================] - 9s 154us/step - loss: 0.0480 - acc: 0.9850 - val_loss: 0.0822 - val_acc: 0.9738 Epoch 4/4 60000/60000 [==============================] - 9s 154us/step - loss: 0.0325 - acc: 0.9900 - val_loss: 0.0693 - val_acc: 0.9796 Baseline Error: 2.04% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 18s 299us/step - loss: 0.2427 - acc: 0.9291 - val_loss: 0.2001 - val_acc: 0.9368 Epoch 2/4 60000/60000 [==============================] - 7s 124us/step - loss: 0.0851 - acc: 0.9748 - val_loss: 0.0984 - val_acc: 0.9698 Epoch 3/4 60000/60000 [==============================] - 7s 124us/step - loss: 0.0524 - acc: 0.9845 - val_loss: 0.0868 - val_acc: 0.9733 Epoch 4/4 60000/60000 [==============================] - 7s 123us/step - loss: 0.0344 - acc: 0.9897 - val_loss: 0.0639 - val_acc: 0.9802 Baseline Error: 1.98% Train on 60000 samples, validate on 10000 samples Epoch 1/4 60000/60000 [==============================] - 17s 285us/step - loss: 0.2732 - acc: 0.9209 - val_loss: 0.1196 - val_acc: 0.9642 Epoch 2/4 60000/60000 [==============================] - 6s 106us/step - loss: 0.0936 - acc: 0.9725 - val_loss: 0.0807 - val_acc: 0.9750 Epoch 3/4 60000/60000 [==============================] - 6s 104us/step - loss: 0.0592 - acc: 0.9821 - val_loss: 0.0699 - val_acc: 0.9784 Epoch 4/4 60000/60000 [==============================] - 6s 105us/step - loss: 0.0387 - acc: 0.9888 - val_loss: 0.0787 - val_acc: 0.9738 Baseline Error: 2.62%
from mpl_toolkits.mplot3d import Axes3D
for optimizer in optimizers:
acc = np.array(accuracy[optimizer]).reshape((5, 5))
fig = plt.figure()
ax = Axes3D(fig)
X = param
Y = batch_size_list
X, Y = np.meshgrid(X, Y)
Z = acc
ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap='CMRmap')
Из графиков видно, что параметр batch_size не оказывает влияние на точность. В дальнейшем будет использоваться только один параметр (число нейронов)
for optimizer in optimizers:
x = np.array(params)
acc = np.array(accuracy[optimizer])
std_acc = acc.std()
plt.plot(x, acc)
plt.fill_between(x, acc+std_acc,
acc-std_acc, facecolor='blue', alpha=0.3)
plt.title('{} accuracy'.format(optimizer))
plt.ylabel('accuracy')
plt.xlabel('param')
plt.show()
Графики показывают, что точность начинает убывать для значения параметра <300 для всех трех алгоритмов. Также видно, что SGD убывает быстрее всего. Высокое стандартное отклонение не позволяет выявить отличия у ADAM и Nesterov Momentum
Здесь исследуется зависимость точности алгоритма ADAM от значения параметра. Рассмотрим 10 значений числа нейронов: от 1 до 50
params = np.linspace(1, 50, num=10, dtype='Int32')
optimizer = 'adam'
accuracy = list()
for param in params:
model = baseline_model(optimizer, param)
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=8, batch_size=1000, verbose=1)
scores = model.evaluate(X_test, y_test, verbose=0)
accuracy.append(scores[1])
print("Baseline Error: %.2f%%" % (100-scores[1]*100))
Train on 60000 samples, validate on 10000 samples Epoch 1/8 60000/60000 [==============================] - 6s 101us/step - loss: 2.2658 - acc: 0.1377 - val_loss: 2.2056 - val_acc: 0.1742 Epoch 2/8 60000/60000 [==============================] - 0s 7us/step - loss: 2.1382 - acc: 0.1855 - val_loss: 2.0781 - val_acc: 0.1872 Epoch 3/8 60000/60000 [==============================] - 0s 6us/step - loss: 2.0399 - acc: 0.1979 - val_loss: 2.0086 - val_acc: 0.2037 Epoch 4/8 60000/60000 [==============================] - 0s 7us/step - loss: 1.9852 - acc: 0.2024 - val_loss: 1.9668 - val_acc: 0.2095 Epoch 5/8 60000/60000 [==============================] - 0s 6us/step - loss: 1.9488 - acc: 0.2064 - val_loss: 1.9364 - val_acc: 0.2122 Epoch 6/8 60000/60000 [==============================] - 0s 6us/step - loss: 1.9214 - acc: 0.2104 - val_loss: 1.9122 - val_acc: 0.2141 Epoch 7/8 60000/60000 [==============================] - 0s 6us/step - loss: 1.8993 - acc: 0.2133 - val_loss: 1.8931 - val_acc: 0.2153 Epoch 8/8 60000/60000 [==============================] - 0s 7us/step - loss: 1.8811 - acc: 0.2172 - val_loss: 1.8767 - val_acc: 0.2196 Baseline Error: 78.04% Train on 60000 samples, validate on 10000 samples Epoch 1/8 60000/60000 [==============================] - 6s 105us/step - loss: 2.1540 - acc: 0.3435 - val_loss: 1.8747 - val_acc: 0.5415 Epoch 2/8 60000/60000 [==============================] - 1s 9us/step - loss: 1.5259 - acc: 0.6302 - val_loss: 1.1882 - val_acc: 0.7021 Epoch 3/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.9920 - acc: 0.7429 - val_loss: 0.8158 - val_acc: 0.7930 Epoch 4/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.7381 - acc: 0.8014 - val_loss: 0.6466 - val_acc: 0.8224 Epoch 5/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.6145 - acc: 0.8271 - val_loss: 0.5563 - val_acc: 0.8407 Epoch 6/8 60000/60000 [==============================] - 1s 9us/step - loss: 0.5447 - acc: 0.8432 - val_loss: 0.5038 - val_acc: 0.8571 Epoch 7/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.5005 - acc: 0.8548 - val_loss: 0.4689 - val_acc: 0.8638 Epoch 8/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.4692 - acc: 0.8645 - val_loss: 0.4438 - val_acc: 0.8747 Baseline Error: 12.53% Train on 60000 samples, validate on 10000 samples Epoch 1/8 60000/60000 [==============================] - 6s 105us/step - loss: 2.0059 - acc: 0.3940 - val_loss: 1.5664 - val_acc: 0.5935 Epoch 2/8 60000/60000 [==============================] - 1s 10us/step - loss: 1.1838 - acc: 0.7068 - val_loss: 0.8478 - val_acc: 0.8129 Epoch 3/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.7113 - acc: 0.8242 - val_loss: 0.5694 - val_acc: 0.8513 Epoch 4/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.5263 - acc: 0.8587 - val_loss: 0.4531 - val_acc: 0.8793 Epoch 5/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.4390 - acc: 0.8809 - val_loss: 0.3926 - val_acc: 0.8931 Epoch 6/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.3896 - acc: 0.8936 - val_loss: 0.3559 - val_acc: 0.9000 Epoch 7/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.3586 - acc: 0.9005 - val_loss: 0.3325 - val_acc: 0.9045 Epoch 8/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.3364 - acc: 0.9061 - val_loss: 0.3144 - val_acc: 0.9110 Baseline Error: 8.90% Train on 60000 samples, validate on 10000 samples Epoch 1/8 60000/60000 [==============================] - 6s 107us/step - loss: 1.8853 - acc: 0.4440 - val_loss: 1.3057 - val_acc: 0.7730 Epoch 2/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.9154 - acc: 0.8164 - val_loss: 0.6358 - val_acc: 0.8567 Epoch 3/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.5479 - acc: 0.8645 - val_loss: 0.4570 - val_acc: 0.8809 Epoch 4/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.4296 - acc: 0.8868 - val_loss: 0.3809 - val_acc: 0.8993 Epoch 5/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.3728 - acc: 0.8986 - val_loss: 0.3411 - val_acc: 0.9081 Epoch 6/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.3358 - acc: 0.9070 - val_loss: 0.3092 - val_acc: 0.9154 Epoch 7/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.3066 - acc: 0.9150 - val_loss: 0.2886 - val_acc: 0.9198 Epoch 8/8 60000/60000 [==============================] - 1s 10us/step - loss: 0.2874 - acc: 0.9201 - val_loss: 0.2750 - val_acc: 0.9210 Baseline Error: 7.90% Train on 60000 samples, validate on 10000 samples Epoch 1/8 60000/60000 [==============================] - 7s 110us/step - loss: 1.7951 - acc: 0.5693 - val_loss: 1.0951 - val_acc: 0.8020 Epoch 2/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.7571 - acc: 0.8337 - val_loss: 0.5294 - val_acc: 0.8723 Epoch 3/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.4704 - acc: 0.8800 - val_loss: 0.3961 - val_acc: 0.8967 Epoch 4/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.3814 - acc: 0.8976 - val_loss: 0.3403 - val_acc: 0.9072 Epoch 5/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.3386 - acc: 0.9069 - val_loss: 0.3094 - val_acc: 0.9141 Epoch 6/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.3129 - acc: 0.9130 - val_loss: 0.2913 - val_acc: 0.9181 Epoch 7/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.2952 - acc: 0.9171 - val_loss: 0.2790 - val_acc: 0.9211 Epoch 8/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.2822 - acc: 0.9206 - val_loss: 0.2674 - val_acc: 0.9241 Baseline Error: 7.59% Train on 60000 samples, validate on 10000 samples Epoch 1/8 60000/60000 [==============================] - 7s 109us/step - loss: 1.6464 - acc: 0.6102 - val_loss: 0.9119 - val_acc: 0.8294 Epoch 2/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.6420 - acc: 0.8560 - val_loss: 0.4626 - val_acc: 0.8895 Epoch 3/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.4202 - acc: 0.8898 - val_loss: 0.3634 - val_acc: 0.9035 Epoch 4/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.3534 - acc: 0.9034 - val_loss: 0.3219 - val_acc: 0.9121 Epoch 5/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.3199 - acc: 0.9107 - val_loss: 0.2976 - val_acc: 0.9169 Epoch 6/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.2973 - acc: 0.9163 - val_loss: 0.2825 - val_acc: 0.9201 Epoch 7/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.2805 - acc: 0.9217 - val_loss: 0.2691 - val_acc: 0.9229 Epoch 8/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.2676 - acc: 0.9245 - val_loss: 0.2587 - val_acc: 0.9262 Baseline Error: 7.38% Train on 60000 samples, validate on 10000 samples Epoch 1/8 60000/60000 [==============================] - 7s 111us/step - loss: 1.6308 - acc: 0.6217 - val_loss: 0.8581 - val_acc: 0.8375 Epoch 2/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.6020 - acc: 0.8612 - val_loss: 0.4327 - val_acc: 0.8906 Epoch 3/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.3985 - acc: 0.8941 - val_loss: 0.3469 - val_acc: 0.9053 Epoch 4/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.3400 - acc: 0.9061 - val_loss: 0.3127 - val_acc: 0.9116 Epoch 5/8 60000/60000 [==============================] - 1s 12us/step - loss: 0.3088 - acc: 0.9136 - val_loss: 0.2898 - val_acc: 0.9183 Epoch 6/8 60000/60000 [==============================] - 1s 12us/step - loss: 0.2866 - acc: 0.9196 - val_loss: 0.2716 - val_acc: 0.9213 Epoch 7/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.2698 - acc: 0.9243 - val_loss: 0.2583 - val_acc: 0.9269 Epoch 8/8 60000/60000 [==============================] - 1s 11us/step - loss: 0.2553 - acc: 0.9284 - val_loss: 0.2466 - val_acc: 0.9297 Baseline Error: 7.03% Train on 60000 samples, validate on 10000 samples Epoch 1/8 60000/60000 [==============================] - 7s 113us/step - loss: 1.5121 - acc: 0.6585 - val_loss: 0.7585 - val_acc: 0.8388 Epoch 2/8 60000/60000 [==============================] - 1s 12us/step - loss: 0.5594 - acc: 0.8644 - val_loss: 0.4194 - val_acc: 0.8925 Epoch 3/8 60000/60000 [==============================] - 1s 12us/step - loss: 0.3874 - acc: 0.8961 - val_loss: 0.3387 - val_acc: 0.9073 Epoch 4/8 60000/60000 [==============================] - 1s 13us/step - loss: 0.3288 - acc: 0.9092 - val_loss: 0.3010 - val_acc: 0.9161 Epoch 5/8 60000/60000 [==============================] - 1s 12us/step - loss: 0.2959 - acc: 0.9177 - val_loss: 0.2761 - val_acc: 0.9227 Epoch 6/8 60000/60000 [==============================] - 1s 12us/step - loss: 0.2721 - acc: 0.9237 - val_loss: 0.2584 - val_acc: 0.9274 Epoch 7/8 60000/60000 [==============================] - 1s 12us/step - loss: 0.2540 - acc: 0.9291 - val_loss: 0.2441 - val_acc: 0.9314 Epoch 8/8 60000/60000 [==============================] - 1s 13us/step - loss: 0.2392 - acc: 0.9329 - val_loss: 0.2323 - val_acc: 0.9325 Baseline Error: 6.75% Train on 60000 samples, validate on 10000 samples Epoch 1/8 60000/60000 [==============================] - 7s 114us/step - loss: 1.4577 - acc: 0.6863 - val_loss: 0.6938 - val_acc: 0.8577 Epoch 2/8 60000/60000 [==============================] - 1s 13us/step - loss: 0.5215 - acc: 0.8724 - val_loss: 0.3952 - val_acc: 0.8998 Epoch 3/8 60000/60000 [==============================] - 1s 13us/step - loss: 0.3690 - acc: 0.9004 - val_loss: 0.3220 - val_acc: 0.9122 Epoch 4/8 60000/60000 [==============================] - 1s 13us/step - loss: 0.3133 - acc: 0.9140 - val_loss: 0.2850 - val_acc: 0.9205 Epoch 5/8 60000/60000 [==============================] - 1s 13us/step - loss: 0.2805 - acc: 0.9224 - val_loss: 0.2601 - val_acc: 0.9263 Epoch 6/8 60000/60000 [==============================] - 1s 12us/step - loss: 0.2575 - acc: 0.9280 - val_loss: 0.2420 - val_acc: 0.9303 Epoch 7/8 60000/60000 [==============================] - 1s 13us/step - loss: 0.2400 - acc: 0.9327 - val_loss: 0.2297 - val_acc: 0.9341 Epoch 8/8 60000/60000 [==============================] - 1s 13us/step - loss: 0.2261 - acc: 0.9362 - val_loss: 0.2163 - val_acc: 0.9374 Baseline Error: 6.26% Train on 60000 samples, validate on 10000 samples Epoch 1/8 60000/60000 [==============================] - 7s 116us/step - loss: 1.4326 - acc: 0.6756 - val_loss: 0.6596 - val_acc: 0.8498 Epoch 2/8 60000/60000 [==============================] - 1s 13us/step - loss: 0.4957 - acc: 0.8753 - val_loss: 0.3837 - val_acc: 0.8980 Epoch 3/8 60000/60000 [==============================] - 1s 13us/step - loss: 0.3595 - acc: 0.9017 - val_loss: 0.3182 - val_acc: 0.9114 Epoch 4/8 60000/60000 [==============================] - 1s 13us/step - loss: 0.3124 - acc: 0.9129 - val_loss: 0.2873 - val_acc: 0.9194 Epoch 5/8 60000/60000 [==============================] - 1s 13us/step - loss: 0.2840 - acc: 0.9206 - val_loss: 0.2670 - val_acc: 0.9244 Epoch 6/8 60000/60000 [==============================] - 1s 13us/step - loss: 0.2637 - acc: 0.9263 - val_loss: 0.2484 - val_acc: 0.9298 Epoch 7/8 60000/60000 [==============================] - 1s 13us/step - loss: 0.2466 - acc: 0.9311 - val_loss: 0.2350 - val_acc: 0.9331 Epoch 8/8 60000/60000 [==============================] - 1s 12us/step - loss: 0.2308 - acc: 0.9357 - val_loss: 0.2229 - val_acc: 0.9362 Baseline Error: 6.38%
x = np.array(params)
acc = np.array(accuracy)
std_acc = acc.std()
plt.plot(x, acc)
plt.fill_between(x, acc+std_acc,
acc-std_acc, facecolor='blue', alpha=0.3)
plt.title('{} accuracy'.format(optimizer))
plt.ylabel('accuracy')
plt.xlabel('param')
plt.show()
Видно, что можно выбрать значение параметра 10 без дначительной потери качества модели
Для данной задачи среди предложенных алгоритмов оптимизации лучшим является ADAM