In this notebook, you'll learn how to use the powerful PyTorch framework to specify a conv net architecture and train it on the CIFAR-10 dataset.
import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable
from torch.utils.data import DataLoader
from torch.utils.data import sampler
import torchvision.datasets as dset
import torchvision.transforms as T
import numpy as np
import timeit
You've written a lot of code in this assignment to provide a whole host of neural network functionality. Dropout, Batch Norm, and 2D convolutions are some of the workhorses of deep learning in computer vision. You've also worked hard to make your code efficient and vectorized.
For the last part of this assignment, though, we're going to leave behind your beautiful codebase and instead migrate to one of two popular deep learning frameworks: in this instance, PyTorch (or TensorFlow, if you switch over to that notebook).
Why?
If you've used Torch before, but are new to PyTorch, this tutorial might be of use: http://pytorch.org/tutorials/beginner/former_torchies_tutorial.html
Otherwise, this notebook will walk you through much of what you need to do to train models in Torch. See the end of the notebook for some links to helpful tutorials if you want to learn more or need further clarification on topics that aren't fully explained here.
We load the CIFAR-10 dataset. This might take a couple minutes the first time you do it, but the files should stay cached after that.
class ChunkSampler(sampler.Sampler):
"""Samples elements sequentially from some offset.
Arguments:
num_samples: # of desired datapoints
start: offset where we should start selecting from
"""
def __init__(self, num_samples, start = 0):
self.num_samples = num_samples
self.start = start
def __iter__(self):
return iter(range(self.start, self.start + self.num_samples))
def __len__(self):
return self.num_samples
NUM_TRAIN = 49000
NUM_VAL = 1000
cifar10_train = dset.CIFAR10('./cs231n/datasets', train=True, download=True,
transform=T.ToTensor())
loader_train = DataLoader(cifar10_train, batch_size=64, sampler=ChunkSampler(NUM_TRAIN, 0))
cifar10_val = dset.CIFAR10('./cs231n/datasets', train=True, download=True,
transform=T.ToTensor())
loader_val = DataLoader(cifar10_val, batch_size=64, sampler=ChunkSampler(NUM_VAL, NUM_TRAIN))
cifar10_test = dset.CIFAR10('./cs231n/datasets', train=False, download=True,
transform=T.ToTensor())
loader_test = DataLoader(cifar10_test, batch_size=64)
Files already downloaded and verified Files already downloaded and verified Files already downloaded and verified
For now, we're going to use a CPU-friendly datatype. Later, we'll switch to a datatype that will move all our computations to the GPU and measure the speedup.
dtype = torch.FloatTensor # the CPU datatype
# Constant to control how frequently we print train loss
print_every = 100
# This is a little utility that we'll use to reset the model
# if we want to re-initialize all our parameters
def reset(m):
if hasattr(m, 'reset_parameters'):
m.reset_parameters()
Let's start by looking at a simple model. First, note that PyTorch operates on Tensors, which are n-dimensional arrays functionally analogous to numpy's ndarrays, with the additional feature that they can be used for computations on GPUs.
We'll provide you with a Flatten function, which we explain here. Remember that our image data (and more relevantly, our intermediate feature maps) are initially N x C x H x W, where:
This is the right way to represent the data when we are doing something like a 2D convolution, that needs spatial understanding of where the intermediate features are relative to each other. When we input data into fully connected affine layers, however, we want each datapoint to be represented by a single vector -- it's no longer useful to segregate the different channels, rows, and columns of the data. So, we use a "Flatten" operation to collapse the C x H x W values per representation into a single long vector. The Flatten function below first reads in the N, C, H, and W values from a given batch of data, and then returns a "view" of that data. "View" is analogous to numpy's "reshape" method: it reshapes x's dimensions to be N x ??, where ?? is allowed to be anything (in this case, it will be C x H x W, but we don't need to specify that explicitly).
class Flatten(nn.Module):
def forward(self, x):
N, C, H, W = x.size() # read in N, C, H, W
return x.view(N, -1) # "flatten" the C * H * W values into a single vector per image
The first step to training your own model is defining its architecture.
Here's an example of a convolutional neural network defined in PyTorch -- try to understand what each line is doing, remembering that each layer is composed upon the previous layer. We haven't trained anything yet - that'll come next - for now, we want you to understand how everything gets set up. nn.Sequential is a container which applies each layer one after the other.
In that example, you see 2D convolutional layers (Conv2d), ReLU activations, and fully-connected layers (Linear). You also see the Cross-Entropy loss function, and the Adam optimizer being used.
Make sure you understand why the parameters of the Linear layer are 5408 and 10.
# Here's where we define the architecture of the model...
simple_model = nn.Sequential(
nn.Conv2d(3, 32, kernel_size=7, stride=2),
nn.ReLU(inplace=True),
Flatten(), # see above for explanation
nn.Linear(5408, 10), # affine layer
)
# Set the type of all data in this model to be FloatTensor
simple_model.type(dtype)
loss_fn = nn.CrossEntropyLoss().type(dtype)
optimizer = optim.Adam(simple_model.parameters(), lr=1e-2) # lr sets the learning rate of the optimizer
PyTorch supports many other layer types, loss functions, and optimizers - you will experiment with these next. Here's the official API documentation for these (if any of the parameters used above were unclear, this resource will also be helpful). One note: what we call in the class "spatial batch norm" is called "BatchNorm2D" in PyTorch.
In this section, we're going to specify a model for you to construct. The goal here isn't to get good performance (that'll be next), but instead to get comfortable with understanding the PyTorch documentation and configuring your own model.
Using the code provided above as guidance, and using the following PyTorch documentation, specify a model with the following architecture:
And finally, set up a cross-entropy loss function and the RMSprop learning rule.
fixed_model_base = nn.Sequential(
nn.Conv2d(3, 32, kernel_size=7, stride=1),
nn.ReLU(inplace=True),
nn.BatchNorm2d(32),
nn.MaxPool2d(2, stride=2),
Flatten(),
nn.Linear(5408, 1024),
nn.ReLU(inplace=True),
nn.Linear(1024, 10)
)
fixed_model = fixed_model_base.type(dtype)
To make sure you're doing the right thing, use the following tool to check the dimensionality of your output (it should be 64 x 10, since our batches have size 64 and the output of the final affine layer should be 10, corresponding to our 10 classes):
## Now we're going to feed a random batch into the model you defined and make sure the output is the right size
x = torch.randn(64, 3, 32, 32).type(dtype)
x_var = Variable(x.type(dtype)) # Construct a PyTorch Variable out of your input data
ans = fixed_model(x_var) # Feed it through the model!
# Check to make sure what comes out of your model
# is the right dimensionality... this should be True
# if you've done everything correctly
np.array_equal(np.array(ans.size()), np.array([64, 10]))
True
Now, we're going to switch the dtype of the model and our data to the GPU-friendly tensors, and see what happens... everything is the same, except we are casting our model and input tensors as this new dtype instead of the old one.
If this returns false, or otherwise fails in a not-graceful way (i.e., with some error message), you may not have an NVIDIA GPU available on your machine. If you're running locally, we recommend you switch to Google Cloud and follow the instructions to set up a GPU there. If you're already on Google Cloud, something is wrong -- make sure you followed the instructions on how to request and use a GPU on your instance. If you did, post on Piazza or come to Office Hours so we can help you debug.
# Verify that CUDA is properly configured and you have a GPU available
torch.cuda.is_available()
True
import copy
gpu_dtype = torch.cuda.FloatTensor
fixed_model_gpu = copy.deepcopy(fixed_model_base).type(gpu_dtype)
x_gpu = torch.randn(64, 3, 32, 32).type(gpu_dtype)
x_var_gpu = Variable(x.type(gpu_dtype)) # Construct a PyTorch Variable out of your input data
ans = fixed_model_gpu(x_var_gpu) # Feed it through the model!
# Check to make sure what comes out of your model
# is the right dimensionality... this should be True
# if you've done everything correctly
np.array_equal(np.array(ans.size()), np.array([64, 10]))
True
Run the following cell to evaluate the performance of the forward pass running on the CPU:
%%timeit
ans = fixed_model(x_var)
10 loops, best of 3: 20 ms per loop
... and now the GPU:
%%timeit
torch.cuda.synchronize() # Make sure there are no pending GPU computations
ans = fixed_model_gpu(x_var_gpu) # Feed it through the model!
torch.cuda.synchronize() # Make sure there are no pending GPU computations
The slowest run took 4.18 times longer than the fastest. This could mean that an intermediate result is being cached 1000 loops, best of 3: 684 µs per loop
You should observe that even a simple forward pass like this is significantly faster on the GPU. So for the rest of the assignment (and when you go train your models in assignment 3 and your project!), you should use the GPU datatype for your model and your tensors: as a reminder that is torch.cuda.FloatTensor (in our notebook here as gpu_dtype)
Now that you've seen how to define a model and do a single forward pass of some data through it, let's walk through how you'd actually train one whole epoch over your training data (using the simple_model we provided above).
Make sure you understand how each PyTorch function used below corresponds to what you implemented in your custom neural network implementation.
Note that because we are not resetting the weights anywhere below, if you run the cell multiple times, you are effectively training multiple epochs (so your performance should improve).
First, set up an RMSprop optimizer (using a 1e-3 learning rate) and a cross-entropy loss function:
loss_fn = nn.CrossEntropyLoss().type(dtype)
optimizer = optim.RMSprop(fixed_model_gpu.parameters(), lr=1e-3)
# This sets the model in "training" mode. This is relevant for some layers that may have different behavior
# in training mode vs testing mode, such as Dropout and BatchNorm.
fixed_model_gpu.train()
# Load one batch at a time.
for t, (x, y) in enumerate(loader_train):
x_var = Variable(x.type(gpu_dtype))
y_var = Variable(y.type(gpu_dtype).long())
# This is the forward pass: predict the scores for each class, for each x in the batch.
scores = fixed_model_gpu(x_var)
# Use the correct y values and the predicted y values to compute the loss.
loss = loss_fn(scores, y_var)
if (t + 1) % print_every == 0:
print('t = %d, loss = %.4f' % (t + 1, loss.data[0]))
# Zero out all of the gradients for the variables which the optimizer will update.
optimizer.zero_grad()
# This is the backwards pass: compute the gradient of the loss with respect to each
# parameter of the model.
loss.backward()
# Actually update the parameters of the model using the gradients computed by the backwards pass.
optimizer.step()
t = 100, loss = 1.4974 t = 200, loss = 1.5574 t = 300, loss = 1.4284 t = 400, loss = 1.2324 t = 500, loss = 1.1979 t = 600, loss = 1.4071 t = 700, loss = 1.1798
Now you've seen how the training process works in PyTorch. To save you writing boilerplate code, we're providing the following helper functions to help you train for multiple epochs and check the accuracy of your model:
def train(model, loss_fn, optimizer, num_epochs=1):
for epoch in range(num_epochs):
print('Starting epoch %d / %d' % (epoch + 1, num_epochs))
model.train()
for t, (x, y) in enumerate(loader_train):
x_var = Variable(x.type(gpu_dtype))
y_var = Variable(y.type(gpu_dtype).long())
scores = model(x_var)
loss = loss_fn(scores, y_var)
if (t + 1) % print_every == 0:
print('t = %d, loss = %.4f' % (t + 1, loss.data[0]))
optimizer.zero_grad()
loss.backward()
optimizer.step()
def check_accuracy(model, loader):
if loader.dataset.train:
print('Checking accuracy on validation set')
else:
print('Checking accuracy on test set')
num_correct = 0
num_samples = 0
model.eval() # Put the model in test mode (the opposite of model.train(), essentially)
for x, y in loader:
x_var = Variable(x.type(gpu_dtype), volatile=True)
scores = model(x_var)
_, preds = scores.data.cpu().max(1)
num_correct += (preds == y).sum()
num_samples += preds.size(0)
acc = float(num_correct) / num_samples
print('Got %d / %d correct (%.2f)' % (num_correct, num_samples, 100 * acc))
Let's see the train and check_accuracy code in action -- feel free to use these methods when evaluating the models you develop below.
You should get a training loss of around 1.2-1.4, and a validation accuracy of around 50-60%. As mentioned above, if you re-run the cells, you'll be training more epochs, so your performance will improve past these numbers.
But don't worry about getting these numbers better -- this was just practice before you tackle designing your own model.
torch.cuda.random.manual_seed(12345)
fixed_model_gpu.apply(reset)
train(fixed_model_gpu, loss_fn, optimizer, num_epochs=1)
check_accuracy(fixed_model_gpu, loader_val)
Starting epoch 1 / 1 t = 100, loss = 1.3117 t = 200, loss = 1.5158 t = 300, loss = 1.4602 t = 400, loss = 1.2262 t = 500, loss = 1.1600 t = 600, loss = 1.3642 t = 700, loss = 1.1776 Checking accuracy on validation set Got 530 / 1000 correct (53.00)
And note that you can use the check_accuracy function to evaluate on either the test set or the validation set, by passing either loader_test or loader_val as the second argument to check_accuracy. You should not touch the test set until you have finished your architecture and hyperparameter tuning, and only run the test set once at the end to report a final value.
Now it's your job to experiment with architectures, hyperparameters, loss functions, and optimizers to train a model that achieves >=70% accuracy on the CIFAR-10 validation set. You can use the check_accuracy and train functions from above.
For each network architecture that you try, you should tune the learning rate and regularization strength. When doing this there are a couple important things to keep in mind:
If you are feeling adventurous there are many other features you can implement to try and improve your performance. You are not required to implement any of these; however they would be good things to try for extra credit.
If you do decide to implement something extra, clearly describe it in the "Extra Credit Description" cell below.
At the very least, you should be able to train a ConvNet that gets at least 70% accuracy on the validation set. This is just a lower bound - if you are careful it should be possible to get accuracies much higher than that! Extra credit points will be awarded for particularly high-scoring models or unique approaches.
You should use the space below to experiment and train your network.
Have fun and happy training!
model = nn.Sequential(
nn.Conv2d(3, 32, kernel_size=3, stride=1, padding=1),
nn.PReLU(),
nn.BatchNorm2d(32),
nn.Dropout(p=0.3),
nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
nn.PReLU(),
nn.BatchNorm2d(64),
nn.Dropout(p=0.3),
nn.Conv2d(64, 64, kernel_size=3, stride=2, padding=1),
nn.PReLU(),
nn.BatchNorm2d(64),
nn.Dropout(p=0.3),
nn.Conv2d(64, 128, kernel_size=3, stride=2, padding=1),
nn.PReLU(),
nn.BatchNorm2d(128),
nn.Dropout(p=0.3),
nn.Conv2d(128, 256, kernel_size=3, stride=2, padding=1),
nn.PReLU(),
nn.BatchNorm2d(256),
nn.Dropout(p=0.3),
nn.Conv2d(256, 512, kernel_size=3, stride=2, padding=1),
nn.PReLU(),
nn.BatchNorm2d(512),
nn.Dropout(p=0.2),
nn.Conv2d(512, 10, kernel_size=2, stride=1, padding=0),
Flatten())
# Train your model here, and make sure the output of this cell is the accuracy of your best model on the
# train, val, and test sets. Here's some code to get you started. The output of this cell should be the training
# and validation accuracy on your best model (measured by validation accuracy).
model = model.type(gpu_dtype)
torch.cuda.random.manual_seed(12345)
model.apply(reset)
loss_fn = nn.CrossEntropyLoss().type(gpu_dtype)
optimizer = optim.Adam(model.parameters(), lr=1e-4)
train(model, loss_fn, optimizer, num_epochs=20)
check_accuracy(model, loader_val)
Starting epoch 1 / 20 t = 100, loss = 1.9702 t = 200, loss = 1.8682 t = 300, loss = 1.8271 t = 400, loss = 1.5565 t = 500, loss = 1.4968 t = 600, loss = 1.6651 t = 700, loss = 1.6352 Starting epoch 2 / 20 t = 100, loss = 1.1895 t = 200, loss = 1.5510 t = 300, loss = 1.3784 t = 400, loss = 1.1549 t = 500, loss = 1.2061 t = 600, loss = 1.4068 t = 700, loss = 1.4867 Starting epoch 3 / 20 t = 100, loss = 1.1208 t = 200, loss = 1.4006 t = 300, loss = 1.2521 t = 400, loss = 1.0672 t = 500, loss = 1.0029 t = 600, loss = 1.0972 t = 700, loss = 1.2795 Starting epoch 4 / 20 t = 100, loss = 1.0439 t = 200, loss = 1.1625 t = 300, loss = 1.2229 t = 400, loss = 0.9312 t = 500, loss = 0.9445 t = 600, loss = 1.0656 t = 700, loss = 1.1557 Starting epoch 5 / 20 t = 100, loss = 0.8455 t = 200, loss = 0.9777 t = 300, loss = 1.0821 t = 400, loss = 0.8810 t = 500, loss = 0.6625 t = 600, loss = 0.9383 t = 700, loss = 0.9930 Starting epoch 6 / 20 t = 100, loss = 0.7664 t = 200, loss = 0.9229 t = 300, loss = 0.9262 t = 400, loss = 0.7381 t = 500, loss = 0.6488 t = 600, loss = 0.8815 t = 700, loss = 0.9345 Starting epoch 7 / 20 t = 100, loss = 0.6567 t = 200, loss = 0.8922 t = 300, loss = 1.0009 t = 400, loss = 0.6937 t = 500, loss = 0.5989 t = 600, loss = 0.8019 t = 700, loss = 0.8151 Starting epoch 8 / 20 t = 100, loss = 0.5910 t = 200, loss = 0.6665 t = 300, loss = 0.7953 t = 400, loss = 0.6709 t = 500, loss = 0.6216 t = 600, loss = 0.7169 t = 700, loss = 0.7098 Starting epoch 9 / 20 t = 100, loss = 0.5466 t = 200, loss = 0.5636 t = 300, loss = 0.7720 t = 400, loss = 0.5966 t = 500, loss = 0.5076 t = 600, loss = 0.6142 t = 700, loss = 0.6798 Starting epoch 10 / 20 t = 100, loss = 0.3667 t = 200, loss = 0.6589 t = 300, loss = 0.6651 t = 400, loss = 0.5933 t = 500, loss = 0.4896 t = 600, loss = 0.5732 t = 700, loss = 0.6654 Starting epoch 11 / 20 t = 100, loss = 0.4222 t = 200, loss = 0.6034 t = 300, loss = 0.6675 t = 400, loss = 0.5592 t = 500, loss = 0.5062 t = 600, loss = 0.5398 t = 700, loss = 0.6673 Starting epoch 12 / 20 t = 100, loss = 0.3814 t = 200, loss = 0.5468 t = 300, loss = 0.5672 t = 400, loss = 0.5343 t = 500, loss = 0.4410 t = 600, loss = 0.5025 t = 700, loss = 0.5182 Starting epoch 13 / 20 t = 100, loss = 0.3264 t = 200, loss = 0.5560 t = 300, loss = 0.6111 t = 400, loss = 0.4178 t = 500, loss = 0.3393 t = 600, loss = 0.4510 t = 700, loss = 0.4927 Starting epoch 14 / 20 t = 100, loss = 0.3161 t = 200, loss = 0.4615 t = 300, loss = 0.5080 t = 400, loss = 0.4246 t = 500, loss = 0.4098 t = 600, loss = 0.3320 t = 700, loss = 0.4687 Starting epoch 15 / 20 t = 100, loss = 0.3793 t = 200, loss = 0.3641 t = 300, loss = 0.4583 t = 400, loss = 0.3211 t = 500, loss = 0.2731 t = 600, loss = 0.3909 t = 700, loss = 0.3901 Starting epoch 16 / 20 t = 100, loss = 0.2056 t = 200, loss = 0.3734 t = 300, loss = 0.4019 t = 400, loss = 0.3230 t = 500, loss = 0.3434 t = 600, loss = 0.3953 t = 700, loss = 0.2618 Starting epoch 17 / 20 t = 100, loss = 0.2631 t = 200, loss = 0.2856 t = 300, loss = 0.4134 t = 400, loss = 0.3303 t = 500, loss = 0.2611 t = 600, loss = 0.2907 t = 700, loss = 0.3506 Starting epoch 18 / 20 t = 100, loss = 0.2134 t = 200, loss = 0.3013 t = 300, loss = 0.4594 t = 400, loss = 0.1915 t = 500, loss = 0.2199 t = 600, loss = 0.2953 t = 700, loss = 0.3093 Starting epoch 19 / 20 t = 100, loss = 0.2090 t = 200, loss = 0.3487 t = 300, loss = 0.3192 t = 400, loss = 0.1612 t = 500, loss = 0.2704 t = 600, loss = 0.2824 t = 700, loss = 0.2945 Starting epoch 20 / 20 t = 100, loss = 0.1939 t = 200, loss = 0.1465 t = 300, loss = 0.3872 t = 400, loss = 0.2110 t = 500, loss = 0.1961 t = 600, loss = 0.1857 t = 700, loss = 0.2983 Checking accuracy on validation set Got 845 / 1000 correct (84.50)
In the cell below you should write an explanation of what you did, any additional features that you implemented, and any visualizations or graphs that you make in the process of training and evaluating your network.
For this CNN, I have chosen a relatively standard architecture: Conv -> PReLU -> BatchNorm. I have doubled the number of filters with every convolutional layer added (except one layer). The initial network was able to overfit quickly the training dataset, thus I have added a dropout layer after each convolutional layer. This improved the accuracy on the validation dataset tremendously. To reduce the spatial dimension of the network I have used convolution layers with a stride of 2 instead of Max Pool layers. The last layer is a convolutional layer with 10 filters, which output directly the network predictions removing the need of a fully-connected layer.
Now that we've gotten a result we're happy with, we test our final model on the test set (which you should store in best_model). This would be the score we would achieve on a competition. Think about how this compares to your validation set accuracy.
best_model = model
check_accuracy(best_model, loader_test)
Checking accuracy on test set Got 8090 / 10000 correct (80.90)
The next assignment will make heavy use of PyTorch. You might also find it useful for your projects.
Here's a nice tutorial by Justin Johnson that shows off some of PyTorch's features, like dynamic graphs and custom NN modules: http://pytorch.org/tutorials/beginner/pytorch_with_examples.html
If you're interested in reinforcement learning for your final project, this is a good (more advanced) DQN tutorial in PyTorch: http://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html