This is similar to the 'NN' notebook, but will use convolutional neural networks instead of normal neural nets on flattened arrays of the data.

In [1]:
# Standard scientific Python imports
import matplotlib.pyplot as plt
%matplotlib inline

# Import datasets, classifiers and performance metrics
from sklearn import datasets, svm, metrics

# The digits dataset
digits = datasets.load_digits()

# The data that we are interested in is made of 8x8 images of digits, let's
# have a look at the first 4 images, stored in the `images` attribute of the
# dataset.  If we were working from image files, we could load them using
# matplotlib.pyplot.imread.  Note that each image must have the same size. For these
# images, we know which digit they represent: it is given in the 'target' of
# the dataset.
In [2]:
images_and_labels = list(zip(digits.images, digits.target))
for index, (image, label) in enumerate(images_and_labels[:4]):
    plt.subplot(2, 4, index + 1)
    plt.axis('off')
    plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
    plt.title('Training: %i' % label)
In [3]:
# To apply a classifier on this data, we need to flatten the image, to
# turn the data in a (samples, feature) matrix:
from keras.utils.np_utils import to_categorical
digits.target_cat = to_categorical(digits.target)

n_samples = len(digits.images)
data = digits.images
X_train = data[:n_samples // 2]
X_test = data[n_samples // 2:]
y_train = digits.target_cat[:n_samples // 2]
y_test = digits.target_cat[n_samples // 2:]
Using TensorFlow backend.
In [4]:
fig, axes = plt.subplots(nrows=3, ncols=5)
for j in range(3):
    for i in range(5):
        axes[j, i].imshow(X_train[j*5 + i])
        axes[j, i].set_xticklabels([])
        axes[j, i].set_yticklabels([])
plt.tight_layout()
In [5]:
X_train2 = X_train.reshape(X_train.shape[0], 1, X_train.shape[1], X_train.shape[2]) #reshape to have depth of 1
X_test2 = X_test.reshape(X_test.shape[0], 1, X_test.shape[1], X_test.shape[2]) #need that for convolutions
X_train2 = X_train2.astype('float32') #make it float
X_test2 = X_test2.astype('float32')
X_train2 /= 16 #scale 0 to 1
X_test2 /= 16
In [6]:
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Conv2D, MaxPooling2D
In [7]:
from keras.callbacks import EarlyStopping
from numpy.random import seed
seed(8)
model = Sequential()
model.add(Conv2D(10, (2, 2), activation='relu', input_shape=X_train2.shape[1:], data_format='channels_first'))
model.add(Conv2D(16, 2, 2, activation='relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(10, activation='softmax'))
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(X_train2, y_train, validation_split=0.3, epochs=50, callbacks=[EarlyStopping(patience=2)])
/Users/muneebalam/anaconda/lib/python3.6/site-packages/ipykernel_launcher.py:6: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(16, (2, 2), activation="relu")`
  
Train on 628 samples, validate on 270 samples
Epoch 1/50
628/628 [==============================] - 0s - loss: 2.2550 - acc: 0.1545 - val_loss: 2.2042 - val_acc: 0.1889
Epoch 2/50
628/628 [==============================] - 0s - loss: 2.1594 - acc: 0.2818 - val_loss: 2.0993 - val_acc: 0.4333
Epoch 3/50
628/628 [==============================] - 0s - loss: 2.0300 - acc: 0.4602 - val_loss: 1.9529 - val_acc: 0.5407
Epoch 4/50
628/628 [==============================] - 0s - loss: 1.8471 - acc: 0.5748 - val_loss: 1.7581 - val_acc: 0.6704
Epoch 5/50
628/628 [==============================] - 0s - loss: 1.6312 - acc: 0.6608 - val_loss: 1.5225 - val_acc: 0.7481
Epoch 6/50
628/628 [==============================] - 0s - loss: 1.3944 - acc: 0.7102 - val_loss: 1.2807 - val_acc: 0.8074
Epoch 7/50
628/628 [==============================] - 0s - loss: 1.1660 - acc: 0.7659 - val_loss: 1.0741 - val_acc: 0.8000
Epoch 8/50
628/628 [==============================] - 0s - loss: 1.0080 - acc: 0.7755 - val_loss: 0.9143 - val_acc: 0.8259
Epoch 9/50
628/628 [==============================] - 0s - loss: 0.8808 - acc: 0.8010 - val_loss: 0.7965 - val_acc: 0.8333
Epoch 10/50
628/628 [==============================] - 0s - loss: 0.7391 - acc: 0.8376 - val_loss: 0.7162 - val_acc: 0.8407
Epoch 11/50
628/628 [==============================] - 0s - loss: 0.6657 - acc: 0.8360 - val_loss: 0.6538 - val_acc: 0.8444
Epoch 12/50
628/628 [==============================] - 0s - loss: 0.5951 - acc: 0.8455 - val_loss: 0.6034 - val_acc: 0.8481
Epoch 13/50
628/628 [==============================] - 0s - loss: 0.5280 - acc: 0.8599 - val_loss: 0.5738 - val_acc: 0.8519
Epoch 14/50
628/628 [==============================] - 0s - loss: 0.5112 - acc: 0.8392 - val_loss: 0.5425 - val_acc: 0.8556
Epoch 15/50
628/628 [==============================] - 0s - loss: 0.4476 - acc: 0.8949 - val_loss: 0.5216 - val_acc: 0.8556
Epoch 16/50
628/628 [==============================] - 0s - loss: 0.4291 - acc: 0.8646 - val_loss: 0.4940 - val_acc: 0.8593
Epoch 17/50
628/628 [==============================] - 0s - loss: 0.3973 - acc: 0.8917 - val_loss: 0.4817 - val_acc: 0.8667
Epoch 18/50
628/628 [==============================] - 0s - loss: 0.4062 - acc: 0.8822 - val_loss: 0.4534 - val_acc: 0.8667
Epoch 19/50
628/628 [==============================] - 0s - loss: 0.3393 - acc: 0.9156 - val_loss: 0.4508 - val_acc: 0.8704
Epoch 20/50
628/628 [==============================] - 0s - loss: 0.3336 - acc: 0.9045 - val_loss: 0.4355 - val_acc: 0.8704
Epoch 21/50
628/628 [==============================] - 0s - loss: 0.2925 - acc: 0.9283 - val_loss: 0.4188 - val_acc: 0.8704
Epoch 22/50
628/628 [==============================] - 0s - loss: 0.3103 - acc: 0.9108 - val_loss: 0.4133 - val_acc: 0.8667
Epoch 23/50
628/628 [==============================] - 0s - loss: 0.2755 - acc: 0.9204 - val_loss: 0.3995 - val_acc: 0.8704
Epoch 24/50
628/628 [==============================] - 0s - loss: 0.2549 - acc: 0.9283 - val_loss: 0.4007 - val_acc: 0.8667
Epoch 25/50
628/628 [==============================] - 0s - loss: 0.2360 - acc: 0.9459 - val_loss: 0.3957 - val_acc: 0.8667
Epoch 26/50
628/628 [==============================] - 0s - loss: 0.2314 - acc: 0.9475 - val_loss: 0.3777 - val_acc: 0.8704
Epoch 27/50
628/628 [==============================] - 0s - loss: 0.2274 - acc: 0.9443 - val_loss: 0.3722 - val_acc: 0.8667
Epoch 28/50
628/628 [==============================] - 0s - loss: 0.2195 - acc: 0.9490 - val_loss: 0.3697 - val_acc: 0.8741
Epoch 29/50
628/628 [==============================] - 0s - loss: 0.2259 - acc: 0.9411 - val_loss: 0.3655 - val_acc: 0.8704
Epoch 30/50
628/628 [==============================] - 0s - loss: 0.1890 - acc: 0.9522 - val_loss: 0.3474 - val_acc: 0.8815
Epoch 31/50
628/628 [==============================] - 0s - loss: 0.1672 - acc: 0.9634 - val_loss: 0.3466 - val_acc: 0.8741
Epoch 32/50
628/628 [==============================] - 0s - loss: 0.2053 - acc: 0.9411 - val_loss: 0.3329 - val_acc: 0.8815
Epoch 33/50
628/628 [==============================] - 0s - loss: 0.1632 - acc: 0.9538 - val_loss: 0.3498 - val_acc: 0.87040.94
Epoch 34/50
628/628 [==============================] - 0s - loss: 0.1667 - acc: 0.9490 - val_loss: 0.3357 - val_acc: 0.8704
Epoch 35/50
628/628 [==============================] - 0s - loss: 0.1625 - acc: 0.9490 - val_loss: 0.3074 - val_acc: 0.8889
Epoch 36/50
628/628 [==============================] - 0s - loss: 0.1655 - acc: 0.9602 - val_loss: 0.3161 - val_acc: 0.8852
Epoch 37/50
628/628 [==============================] - 0s - loss: 0.1612 - acc: 0.9554 - val_loss: 0.3046 - val_acc: 0.8926
Epoch 38/50
628/628 [==============================] - 0s - loss: 0.1368 - acc: 0.9713 - val_loss: 0.3148 - val_acc: 0.8889
Epoch 39/50
628/628 [==============================] - 0s - loss: 0.1458 - acc: 0.9634 - val_loss: 0.3074 - val_acc: 0.9000
Epoch 40/50
628/628 [==============================] - 0s - loss: 0.1386 - acc: 0.9618 - val_loss: 0.2987 - val_acc: 0.8926
Epoch 41/50
628/628 [==============================] - 0s - loss: 0.1314 - acc: 0.9634 - val_loss: 0.3013 - val_acc: 0.8889
Epoch 42/50
628/628 [==============================] - 0s - loss: 0.1325 - acc: 0.9697 - val_loss: 0.3175 - val_acc: 0.8630
Epoch 43/50
628/628 [==============================] - 0s - loss: 0.1371 - acc: 0.9602 - val_loss: 0.3002 - val_acc: 0.8889
Out[7]:
<keras.callbacks.History at 0x12433a828>
In [8]:
predictions = model.predict(X_test2)
# The classification report required 1D arrays, so I'll convert everything to max probability
from numpy import argmax
predictions1D = argmax(predictions, axis=1) #should have shape (899,)
y_test1D = argmax(y_test, axis=1)
report = metrics.classification_report(y_test1D, predictions1D)

import pandas as pd #for formatting
df = pd.DataFrame([x.split() for x in report.replace('/', '\n').split('\n')]) #have a problematic slash in there
from numpy import concatenate, array
df.columns = concatenate([array(['Digit']), df.iloc[0,:-1]])
df.drop(df.index[:2], inplace=True)
df.set_index('Digit', inplace=True)
df.drop(df.index[pd.isnull(df.index.values)], inplace=True)
df
Out[8]:
precision recall f1-score support
Digit
0 0.99 0.98 0.98 88
1 0.90 0.73 0.80 91
2 0.92 0.90 0.91 86
3 0.90 0.86 0.88 91
4 0.96 0.95 0.95 92
5 0.93 0.97 0.95 91
6 0.94 0.98 0.96 91
7 0.94 0.85 0.89 89
8 0.74 0.82 0.78 88
9 0.81 0.96 0.88 92
avg None None None None
total 0.90 0.90 0.90 899

This looks like it stabilized around 90% accuracy.

In [9]:
sum(predictions1D == y_test1D)/len(y_test1D)
Out[9]:
0.8976640711902113

This is a little better than the other neural network but still much worse than, say, SVM, which got to 97%.