:label:sec_channels
While we have described the multiple channels that comprise each image (e.g., color images have the standard RGB channels to indicate the amount of red, green and blue), until now, we simplified all of our numerical examples by working with just a single input and a single output channel. This has allowed us to think of our inputs, convolutional kernels, and outputs each as two-dimensional arrays.
When we add channels into the mix, our inputs and hidden representations both become three-dimensional arrays. For example, each RGB input image has shape $3\times h\times w$. We refer to this axis, with a size of 3, as the channel dimension. In this section, we will take a deeper look at convolution kernels with multiple input and multiple output channels.
When the input data contains multiple channels, we need to construct a convolution kernel with the same number of input channels as the input data, so that it can perform cross-correlation with the input data. Assuming that the number of channels for the input data is $c_i$, the number of input channels of the convolution kernel also needs to be $c_i$. If our convolution kernel's window shape is $k_h\times k_w$, then when $c_i=1$, we can think of our convolution kernel as just a two-dimensional array of shape $k_h\times k_w$.
However, when $c_i>1$, we need a kernel that contains an array of shape $k_h\times k_w$ for each input channel. Concatenating these $c_i$ arrays together yields a convolution kernel of shape $c_i\times k_h\times k_w$. Since the input and convolution kernel each have $c_i$ channels, we can perform a cross-correlation operation on the two-dimensional array of the input and the two-dimensional kernel array of the convolution kernel for each channel, adding the $c_i$ results together (summing over the channels) to yield a two-dimensional array. This is the result of a two-dimensional cross-correlation between multi-channel input data and a multi-input channel convolution kernel.
In :numref:fig_conv_multi_in
, we demonstrate an example
of a two-dimensional cross-correlation with two input channels.
The shaded portions are the first output element
as well as the input and kernel array elements used in its computation:
$(1\times1+2\times2+4\times3+5\times4)+(0\times0+1\times1+3\times2+4\times3)=56$.
:label:fig_conv_multi_in
To make sure we really understand what is going on here,
we can implement cross-correlation operations with multiple input channels ourselves.
Notice that all we are doing is performing one cross-correlation operation
per channel and then adding up the results using the sum()
function.
Let's import the libraries before jumping into the concept of multiple input and output channels.
%load ../utils/djl-imports
NDManager manager = NDManager.newBaseManager();
public NDArray corr2D(NDArray X, NDArray K) {
long h = K.getShape().get(0);
long w = K.getShape().get(1);
NDArray Y = manager.zeros(new Shape(X.getShape().get(0) - h + 1, X.getShape().get(1) - w + 1));
for (int i = 0; i < Y.getShape().get(0); i++) {
for (int j = 0; j < Y.getShape().get(1); j++) {
NDArray temp = X.get(i + ":" + (i + h) + "," + j + ":" + (j + w)).mul(K);
Y.set(new NDIndex(i + "," + j), temp.sum());
}
}
return Y;
}
public NDArray corr2dMultiIn(NDArray X, NDArray K) {
long h = K.getShape().get(0);
long w = K.getShape().get(1);
// First, traverse along the 0th dimension (channel dimension) of `X` and
// `K`. Then, add them together
NDArray res = manager.zeros(new Shape(X.getShape().get(0) - h + 1, X.getShape().get(1) - w + 1));
for (int i = 0; i < X.getShape().get(0); i++) {
for (int j = 0; j < K.getShape().get(0); j++) {
if (i == j) {
res = res.add(corr2D(X.get(new NDIndex(i)), K.get(new NDIndex(j))));
}
}
}
return res;
}
We can construct the input array X
and the kernel array K
corresponding to the values in the above diagram
to validate the output of the cross-correlation operation.
NDArray X = manager.create(new Shape(2, 3, 3), DataType.INT32);
X.set(new NDIndex(0), manager.arange(9));
X.set(new NDIndex(1), manager.arange(1, 10));
X = X.toType(DataType.FLOAT32, true);
NDArray K = manager.create(new Shape(2, 2, 2), DataType.INT32);
K.set(new NDIndex(0), manager.arange(4));
K.set(new NDIndex(1), manager.arange(1, 5));
K = K.toType(DataType.FLOAT32, true);
corr2dMultiIn(X, K);
Regardless of the number of input channels, so far we always ended up with one output channel. However, as we discussed earlier, it turns out to be essential to have multiple channels at each layer. In the most popular neural network architectures, we actually increase the channel dimension as we go higher up in the neural network, typically downsampling to trade off spatial resolution for greater channel depth. Intuitively, you could think of each channel as responding to some different set of features. Reality is a bit more complicated than the most naive interpretations of this intuition since representations are not learned independent but are rather optimized to be jointly useful. So it may not be that a single channel learns an edge detector but rather that some direction in channel space corresponds to detecting edges.
Denote by $c_i$ and $c_o$ the number of input and output channels, respectively, and let $k_h$ and $k_w$ be the height and width of the kernel. To get an output with multiple channels, we can create a kernel array of shape $c_i\times k_h\times k_w$ for each output channel. We concatenate them on the output channel dimension, so that the shape of the convolution kernel is $c_o\times c_i\times k_h\times k_w$. In cross-correlation operations, the result on each output channel is calculated from the convolution kernel corresponding to that output channel and takes input from all channels in the input array.
We implement a cross-correlation function to calculate the output of multiple channels as shown below.
public NDArray corrMultiInOut(NDArray X, NDArray K) {
long cin = K.getShape().get(0);
long h = K.getShape().get(2);
long w = K.getShape().get(3);
// Traverse along the 0th dimension of `K`, and each time, perform
// cross-correlation operations with input `X`. All of the results are
// merged together using the stack function
NDArray res = manager.create(new Shape(cin, X.getShape().get(1) - h + 1, X.getShape().get(2) - w + 1));
for (int j = 0; j < K.getShape().get(0); j++) {
res.set(new NDIndex(j), corr2dMultiIn(X, K.get(new NDIndex(j))));
}
return res;
}
We construct a convolution kernel with 3 output channels
by concatenating the kernel array K
with K+1
(plus one for each element in K
) and K+2
.
K = NDArrays.stack(new NDList(K, K.add(1), K.add(2)));
K.getShape()
Below, we perform cross-correlation operations
on the input array X
with the kernel array K
.
Now the output contains 3 channels.
The result of the first channel is consistent
with the result of the previous input array X
and the multi-input channel,
single-output channel kernel.
corrMultiInOut(X, K);
At first, a $1 \times 1$ convolution, i.e., $k_h = k_w = 1$, does not seem to make much sense. After all, a convolution correlates adjacent pixels. A $1 \times 1$ convolution obviously does not. Nonetheless, they are popular operations that are sometimes included in the designs of complex deep networks. Let us see in some detail what it actually does.
Because the minimum window is used, the $1\times 1$ convolution loses the ability of larger convolutional layers to recognize patterns consisting of interactions among adjacent elements in the height and width dimensions. The only computation of the $1\times 1$ convolution occurs on the channel dimension.
:numref:fig_conv_1x1
shows the cross-correlation computation
using the $1\times 1$ convolution kernel
with 3 input channels and 2 output channels.
Note that the inputs and outputs have the same height and width.
Each element in the output is derived
from a linear combination of elements at the same position
in the input image.
You could think of the $1\times 1$ convolutional layer
as constituting a fully-connected layer applied at every single pixel location
to transform the $c_i$ corresponding input values into $c_o$ output values.
Because this is still a convolutional layer,
the weights are tied across pixel location.
Thus the $1\times 1$ convolutional layer requires $c_o\times c_i$ weights
(plus the bias terms).
:label:fig_conv_1x1
Let us check whether this works in practice: we implement the $1 \times 1$ convolution using a fully-connected layer. The only thing is that we need to make some adjustments to the data shape before and after the matrix multiplication.
public NDArray corr2dMultiInOut1x1(NDArray X, NDArray K) {
long channelIn = X.getShape().get(0);
long height = X.getShape().get(1);
long width = X.getShape().get(2);
long channelOut = K.getShape().get(0);
X = X.reshape(channelIn, height * width);
K = K.reshape(channelOut, channelIn);
NDArray Y = K.dot(X); // Matrix multiplication in the fully connected layer
return Y.reshape(channelOut, height, width);
}
When performing $1\times 1$ convolution,
the above function is equivalent to the previously implemented cross-correlation function corrMultiInOut()
.
Let us check this with some reference data.
X = manager.randomUniform(0f, 1.0f, new Shape(3, 3, 3));
K = manager.randomUniform(0f, 1.0f, new Shape(2, 3, 1, 1));
NDArray Y1 = corr2dMultiInOut1x1(X, K);
NDArray Y2 = corrMultiInOut(X, K);
System.out.println(Math.abs(Y1.sum().getFloat() - Y2.sum().getFloat()) < 1e-6);
Y1
and Y2
in the last example of this section exactly the same? Why?