Duke Community Standard: By typing your name below, you are certifying that you have adhered to the Duke Community Standard in completing this assignment.
Name: [YOUR_NAME_HERE]
Now that you've run through a simple logistic regression model on MNIST, let's see if we can do better (Hint: we can). For this assignment, you'll build a multilayer perceptron (MLP) and a convolutional neural network (CNN), two popular types of neural networks, and compare their performance. Some potentially useful code:
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
# Import data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
# Helper functions for creating weight variables
def weight_variable(shape):
"""weight_variable generates a weight variable of a given shape."""
initial = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial)
def bias_variable(shape):
"""bias_variable generates a bias variable of a given shape."""
initial = tf.constant(0.1, shape=shape)
return tf.Variable(initial)
# Tensorflow Functions that might also be of interest:
# tf.nn.sigmoid()
# tf.nn.relu()
Build a multilayer perceptron for MNIST digit classfication. Feel free to play around with the model architecture and see how the training time/performance changes, but to begin, try the following:
Image -> fully connected (500 hidden units) -> nonlinearity (Sigmoid/ReLU) -> fully connected (10 hidden units) -> softmax
Skeleton framework for you to fill in (Code you need to provide is marked by ###
):
# Model Inputs
x = ### MNIST images enter graph here ###
y_ = ### MNIST labels enter graph here ###
# Define the graph
### Create your MLP here##
### Make sure to name your MLP output as y_mlp ###
# Loss
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_mlp))
# Optimizer
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
# Evaluation
correct_prediction = tf.equal(tf.argmax(y_mlp, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
with tf.Session() as sess:
# Initialize all variables
sess.run(tf.global_variables_initializer())
# Training regimen
for i in range(4000):
# Validate every 250th batch
if i % 250 == 0:
validation_accuracy = 0
for v in range(10):
batch = mnist.validation.next_batch(100)
validation_accuracy += (1/10) * accuracy.eval(feed_dict={x: batch[0], y_: batch[1]})
print('step %d, validation accuracy %g' % (i, validation_accuracy))
# Train
batch = mnist.train.next_batch(50)
train_step.run(feed_dict={x: batch[0], y_: batch[1]})
print('test accuracy %g' % accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels}))
How do the sigmoid and rectified linear unit (ReLU) compare?
[Your answer here]
Build a simple 2-layer CNN for MNIST digit classfication. Feel free to play around with the model architecture and see how the training time/performance changes, but to begin, try the following:
Image -> CNN (32 5x5 filters) -> nonlinearity (ReLU) -> (2x2 max pool) -> CNN (64 5x5 filters) -> nonlinearity (ReLU) -> (2x2 max pool) -> fully connected (1024 hidden units) -> nonlinearity (ReLU) -> fully connected (10 hidden units) -> softmax
Some additional functions that you might find helpful:
# Convolutional neural network functions
def conv2d(x, W):
"""conv2d returns a 2d convolution layer with full stride."""
return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
def max_pool_2x2(x):
"""max_pool_2x2 downsamples a feature map by 2X."""
return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
# Tensorflow Function that might also be of interest:
# tf.reshape()
Skeleton framework for you to fill in (Code you need to provide is marked by ###
):
Hint: Convolutional Neural Networks are spatial models. You'll want to transform the flattened MNIST vectors into images for the CNN. Similarly, you might want to flatten it again sometime before you do a softmax. You also might want to look into the conv2d() documentation to see what shape kernel/filter it's expecting.
# Model Inputs
x = ### MNIST images enter graph here ###
y_ = ### MNIST labels enter graph here ###
# Define the graph
### Create your CNN here##
### Make sure to name your CNN output as y_conv ###
# Loss
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))
# Optimizer
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
# Evaluation
correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
with tf.Session() as sess:
# Initialize all variables
sess.run(tf.global_variables_initializer())
# Training regimen
for i in range(10000):
# Validate every 250th batch
if i % 250 == 0:
validation_accuracy = 0
for v in range(10):
batch = mnist.validation.next_batch(50)
validation_accuracy += (1/10) * accuracy.eval(feed_dict={x: batch[0], y_: batch[1]})
print('step %d, validation accuracy %g' % (i, validation_accuracy))
# Train
batch = mnist.train.next_batch(50)
train_step.run(feed_dict={x: batch[0], y_: batch[1]})
print('test accuracy %g' % accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels}))
Some differences from the logistic regression model to note:
The CNN model might take a while to train. Depending on your machine, you might expect this to take up to half an hour. If you see your validation performance start to plateau, you can kill the training.
The logistic regression model we used previously was pretty basic, and as such, we were able to get away with using the GradientDescentOptimizer, which performs implements the gradient descent algorithm. For more difficult optimization spaces (such as the ones deep networks pose), we might want to use more sophisticated algorithms. Prof David Carlson has a lecture on this later.
Because of the larger size of our network, notice that our minibatch size has shrunk.
We've added a validation step every 250 minibatches. This let's us see how our model is doing during the training process, rather than sit around twiddling our thumbs and hoping for the best when training finishes. This becomes especially significant as training regimens start approaching days and weeks in length. Normally, we validate on the entire validation set, but for the sake of time we'll just stick to 10 validation minibatches (500 images) for this homework assignment.
How do the MLP and CNN compare in accuracy? Training time? Why would you use one vs the other? Is there a problem you see with MLPs when applied to other image datasets?
[Your answer here]