First, we'll want to import the keras modules we'll be using for our neural network and the numpy and matplotlib modules that we'll be using for displaying our test images.
from keras.models import Sequential
from keras.layers import Conv2D
import numpy
from matplotlib.pyplot import imshow
# tell matplotlib to display images within this notebook
%matplotlib inline
Using TensorFlow backend.
Next, let's set up the structure of our model. We'll start with a really simple model, with just one convolutional layer that has just one filter. We are going to be using 9x9-pixel grayscale images, so we set the input shape accordingly. If we were using color images with red-green-blue channels, the last dimension would be size three (one for each color) instead of one.
kernel_size = 3
image_size = 9
model0 = Sequential()
model0.add(Conv2D(filters=1,
kernel_size=kernel_size,
strides=1,
input_shape=(image_size, image_size, 1)))
Normally at this point, we would compile and train (aka fit) our model, but instead we're going to set the weights manually and then see the output we get on some test images.
First, let's take a look at what the randomly generated weights look like, to understand the format that we'll need to use to set the new weights. By changing the parameters of the model above and looking at how it affects the weight structure, we can understand what each weight is connected to (try it!).
weights = model0.get_weights()
weights
[array([[[[-0.10406637]], [[ 0.5136833 ]], [[ 0.38653535]]], [[[-0.19607425]], [[ 0.42464244]], [[-0.4395233 ]]], [[[ 0.47101247]], [[ 0.16527855]], [[-0.45755833]]]], dtype=float32), array([0.], dtype=float32)]
Now, we change the weights so that the filter will capture a certain pattern. We'll explore more about what this means below, but feel free to start generating some guesses.
layer_num = 0
filter_num = 0
y = 0
for x in range(kernel_size):
weights[layer_num][y][x][0][filter_num] = 1
for y in range(1,kernel_size):
for x in range(kernel_size):
weights[layer_num][y][x][0][filter_num] = -1
weights
[array([[[[ 1.]], [[ 1.]], [[ 1.]]], [[[-1.]], [[-1.]], [[-1.]]], [[[-1.]], [[-1.]], [[-1.]]]], dtype=float32), array([0.], dtype=float32)]
And save those weights back into the model.
model0.set_weights(weights)
Now, let's create some 9x9 images that we will run through our model.
image0 = numpy.array([
[128, 0, 128, 255, 128, 0, 128, 255, 128],
[128, 0, 128, 255, 128, 0, 128, 255, 128],
[128, 0, 128, 255, 128, 0, 128, 255, 128],
[128, 0, 128, 255, 128, 0, 128, 255, 128],
[128, 0, 128, 255, 128, 0, 128, 255, 128],
[128, 0, 128, 255, 128, 0, 128, 255, 128],
[128, 0, 128, 255, 128, 0, 128, 255, 128],
[128, 0, 128, 255, 128, 0, 128, 255, 128],
[128, 0, 128, 255, 128, 0, 128, 255, 128],
], dtype=numpy.uint8)
imshow(image0, cmap='gray')
<matplotlib.image.AxesImage at 0x13873ab10>
image1 = numpy.array([
[128, 128, 128, 128, 128, 128, 128, 128, 128],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[128, 128, 128, 128, 128, 128, 128, 128, 128],
[255, 255, 255, 255, 255, 255, 255, 255, 255],
[128, 128, 128, 128, 128, 128, 128, 128, 128],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[128, 128, 128, 128, 128, 128, 128, 128, 128],
[255, 255, 255, 255, 255, 255, 255, 255, 255],
[128, 128, 128, 128, 128, 128, 128, 128, 128],
], dtype=numpy.uint8)
imshow(image1, cmap='gray')
<matplotlib.image.AxesImage at 0x1388ac110>
The images need to be in a slightly different format for Keras than they do for the imshow command. Right now, they are 9x9 arrays, and we need them to be 9x9x1 -- three dimensional instead of two.
images = []
for image in [image0, image1]: # You may find it easier to take one of these out, to look at them one at a time
images.append(numpy.resize(image, (image_size, image_size, 1)))
And now we give these images to our model and take a look at what the filter has found.
model0.predict(numpy.array(images))
array([[[[ -256.], [ -383.], [ -511.], [ -383.], [ -256.], [ -383.], [ -511.]], [[ -256.], [ -383.], [ -511.], [ -383.], [ -256.], [ -383.], [ -511.]], [[ -256.], [ -383.], [ -511.], [ -383.], [ -256.], [ -383.], [ -511.]], [[ -256.], [ -383.], [ -511.], [ -383.], [ -256.], [ -383.], [ -511.]], [[ -256.], [ -383.], [ -511.], [ -383.], [ -256.], [ -383.], [ -511.]], [[ -256.], [ -383.], [ -511.], [ -383.], [ -256.], [ -383.], [ -511.]], [[ -256.], [ -383.], [ -511.], [ -383.], [ -256.], [ -383.], [ -511.]]], [[[ 0.], [ 0.], [ 0.], [ 0.], [ 0.], [ 0.], [ 0.]], [[-1149.], [-1149.], [-1149.], [-1149.], [-1149.], [-1149.], [-1149.]], [[ -765.], [ -765.], [ -765.], [ -765.], [ -765.], [ -765.], [ -765.]], [[ 381.], [ 381.], [ 381.], [ 381.], [ 381.], [ 381.], [ 381.]], [[ 0.], [ 0.], [ 0.], [ 0.], [ 0.], [ 0.], [ 0.]], [[-1149.], [-1149.], [-1149.], [-1149.], [-1149.], [-1149.], [-1149.]], [[ -765.], [ -765.], [ -765.], [ -765.], [ -765.], [ -765.], [ -765.]]]], dtype=float32)
There are a lot of numbers in the output above: 2 arrays of 7 arrays of 7 arrays of a single element. Why are they in groups of seven?
When we created the model, we asked it to have one filter. In which image do we get the highest absolute values in the filter outputs? How does this relate to the pattern of weights that was set?
Let's move to a slightly more complex model. Now, there are two convolutional layers, the first with two filters and the second with one filter. One other difference is that we're going to be taking strides so that we only examine each pixel once, instead of looking at overlapping groups. This makes it a little simpler to understand the manual weights.
model1 = Sequential()
model1.add(Conv2D(filters=2,
kernel_size=kernel_size,
strides=(3,3),
input_shape=(image_size, image_size, 1)))
model1.add(Conv2D(filters=1, kernel_size=kernel_size))
With a different model structure, we will have a different number of weights to fit.
weights = model1.get_weights()
weights
[array([[[[ 0.18620053, -0.0254904 ]], [[-0.34455183, 0.07090899]], [[ 0.31497756, -0.04884526]]], [[[ 0.34851035, 0.44877866]], [[-0.11065054, 0.39561316]], [[-0.01922548, -0.08240607]]], [[[-0.3907231 , -0.23695081]], [[-0.30338448, 0.34737644]], [[ 0.44047478, 0.09450045]]]], dtype=float32), array([0., 0.], dtype=float32), array([[[[-0.33926663], [-0.37579873]], [[-0.447349 ], [ 0.15102217]], [[-0.19185525], [-0.3894932 ]]], [[[-0.13201055], [ 0.20396301]], [[-0.12954187], [-0.18424916]], [[-0.13625324], [ 0.21633765]]], [[[-0.32364768], [-0.24014083]], [[ 0.25038263], [ 0.156149 ]], [[ 0.09394589], [-0.27343506]]]], dtype=float32), array([0.], dtype=float32)]
As before, we manually set the weights to match some specific patterns.
layer_num = 0
filter_num = 0
for y in range(kernel_size):
for x in range(kernel_size):
if y == x:
weights[layer_num][y][x][0][filter_num] = 1
else:
weights[layer_num][y][x][0][filter_num] = -1
filter_num = 1
for y in range(kernel_size):
for x in range(kernel_size):
if kernel_size - 1 - y == x:
weights[layer_num][y][x][0][filter_num] = 1
else:
weights[layer_num][y][x][0][filter_num] = -1
# layer 1 is for setting the biases of the first layer.
# They are zero by default, so leave them that way and
# move on to layer 2, which contains the weights for the
# filters of the second layer.
layer_num = 2
filter_num = 0
for y in range(kernel_size):
for x in range(kernel_size):
input_filter_num = 0
if y == x:
weights[layer_num][y][x][input_filter_num][filter_num] = 1
else:
weights[layer_num][y][x][input_filter_num][filter_num] = -0.25
input_filter_num = 1
if kernel_size - 1 - y == x:
weights[layer_num][y][x][input_filter_num][filter_num] = 1
else:
weights[layer_num][y][x][input_filter_num][filter_num] = -0.25
weights
[array([[[[ 1., -1.]], [[-1., -1.]], [[-1., 1.]]], [[[-1., -1.]], [[ 1., 1.]], [[-1., -1.]]], [[[-1., 1.]], [[-1., -1.]], [[ 1., -1.]]]], dtype=float32), array([0., 0.], dtype=float32), array([[[[ 1. ], [-0.25]], [[-0.25], [-0.25]], [[-0.25], [ 1. ]]], [[[-0.25], [-0.25]], [[ 1. ], [ 1. ]], [[-0.25], [-0.25]]], [[[-0.25], [ 1. ]], [[-0.25], [-0.25]], [[ 1. ], [-0.25]]]], dtype=float32), array([0.], dtype=float32)]
And save the weights back into the model.
model1.set_weights(weights)
Again, run our test images through the model to see what the filters output.
def predict_images(images):
resized_images = []
for image in images:
resized_images.append(numpy.resize(image, (image_size, image_size, 1)))
return model1.predict(numpy.array(resized_images))
predict_images([image0, image1])
array([[[[-1150.]]], [[[-1150.]]]], dtype=float32)
Note above that neither image0 nor image1 gets a positive output. Create some images that do get positive ouputs from this model. The code below might help you get started.
image_black = numpy.array([
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
], dtype=numpy.uint8)
imshow(image_black, cmap='gray', vmin=0, vmax=255)
<matplotlib.image.AxesImage at 0x138962710>
image_white = numpy.array([
[255, 255, 255, 255, 255, 255, 255, 255, 255],
[255, 255, 255, 255, 255, 255, 255, 255, 255],
[255, 255, 255, 255, 255, 255, 255, 255, 255],
[255, 255, 255, 255, 255, 255, 255, 255, 255],
[255, 255, 255, 255, 255, 255, 255, 255, 255],
[255, 255, 255, 255, 255, 255, 255, 255, 255],
[255, 255, 255, 255, 255, 255, 255, 255, 255],
[255, 255, 255, 255, 255, 255, 255, 255, 255],
[255, 255, 255, 255, 255, 255, 255, 255, 255],
], dtype=numpy.uint8)
imshow(image_white, cmap='gray', vmin=0, vmax=255)
<matplotlib.image.AxesImage at 0x1390262d0>
predict_images([image_black, image_white])
array([[[[ 0.]]], [[[-2295.]]]], dtype=float32)