[Hands-on deep learning-notes] From fully connected layer to convolution

A principle of object recognition - invariance (invariance)

translation invariance

No matter where in the image the detected object appears, the first few layers of the neural network should have a similar response to the same image region, which is "translation invariance".

locality

The first few layers of the neural network should only explore local areas in the input image, and not pay too much attention to the relationship between distant areas in the image. This is the principle of "locality". Eventually, these local features can be aggregated to make predictions at the whole image level.

Let me give you my point of view first:
Convolution is an improvement to full connection in order to satisfy invariance

Limitations of Fully Connected Layers

too many parameters

For high-dimensional perception data, networks with only fully connected layers may become impractical.

Assuming we have a sufficient photo dataset, the dataset contains labeled photos, and each photo has millions of pixels, which means that each input of the network has a million dimensions. Even if the hidden layer dimension is reduced to 1000, this fully connected layer will have $10^6\times 10^3=10^9$ parameters. Trying to train this model would not be feasible, because it would require a large number of GPUs, experience in distributed optimization training, and extraordinary patience.

weak immutability

Let's first take a look at what the fully connected layer looks like when processing two-dimensional data

For one-dimensional input, the fully connected layer works like this:
$$\begin{array}{r}\begin{array}{rl}{\left[\mathbf{H}\right]}_i& =[\mathbf{U}{]}_i+\sum _j[\mathsf{W}{]}_{i,j}[\mathbf{X}{]}_j\end{array}\end{array}$$

In order to preserve the spatial information, we now directly input a 2D data.

So now the input is two-dimensional data containing width and height information, and the output is also two-dimensional. Correspondingly, the weight matrix of the fully connected layer will also increase the two dimensions of width and height to become a 4-dimensional weight matrix.

Example to understand this four-dimensional weight matrix structure
Suppose our input dimension is $20\ast 20$ , need to get a$15\ast$The output of$15$
then this$[W{]}_{15,15,20,20}$is a $15\ast A\; large\; matrix\; of\; 15$ , and each element of this large matrix is$20\ast$A small matrix of$20$
To calculate the output$(3,4)$ , take the large matrix$(3,4)$ The small matrix at the place, and the input perform an element-by-element weighted summation, and then add the bias matrix$(3,4)$ The offset value obtained can be output.

Then similarly, the fully connected layer that processes two-dimensional input can be expressed as:
$$\begin{array}{r}\begin{array}{rl}{\left[\mathbf{H}\right]}_{i,j}& =[\mathbf{U}{]}_{i,j}+\sum _k\sum _l[\mathsf{W}{]}_{i,j,k,l}[\mathbf{X}{]}_{k,l}\end{array}\end{array}$$
$[X{]}_{k,l}$Represents the input $(k,l)$ Pixels at position
$[\mathbf{H}{]}_{i,j}$Represents the output $(i,j)$ pixel
$[\mathbf{U}{]}_{i,j}$is the output to $(i,j)$ The offset added at the position

Next, perform a subscript transformation , let $k=i+a$、$l=j+b$ , then:

$$\begin{array}{r}\begin{array}{rl}{\left[\mathbf{H}\right]}_{i,j}& =[\mathbf{U}{]}_{i,j}+\sum _k\sum _l[\mathsf{W}{]}_{i,j,k,l}[\mathbf{X}{]}_{k,l}\\ & =[U{]}_{i,j}+\sum _a\sum _b[\mathsf{V}{]}_{i,j,a,b}[\mathbf{X}{]}_{i+a,j+b}.\end{array}\end{array}$$
$VsW$ \mathbf{ W}Reindexing of$W$$[X{]}_{i,j}$As the center pixel, by changing the offset $a$ and offset$The\; value\; of\; b$ , accessing$\left[X\right]$ anywhere

Through this form, we can see that if the coordinates of the center pixel change, the weight V responsible for the calculation will also change, which violates the translation invariance, and V should not depend on the coordinates of the center pixel .
At the same time, by changing the offset $a$ and offset$The\; value\; of\; b$ , accessing$\left[X\right]$ , which alsolocal invariance.

Translation invariance adjustment

We should keep the weight V responsible for the calculation constant when the position of the center pixel changes:
$$[\mathbf{V}{]}_{i,j,a,b}=[V{]}_{a,b}$$
After that, the eccentric fixed dwell, from
$$[\mathbf{H}{]}_{i,j}=u+\sum _a\sum _b[V{]}_{a,b}[\mathbf{X}{]}_{i+a,j+b}.In$$
this way, each$[\mathbf{X}{]}_{i,j}$When used as a central pixel, the pixels around it will be calculated with the same weight matrix, which is convolution

local invariance adjustment

给$a$、$The\; change\; of\; b$ sets a limit that it does not deviate to a position ( i , j ) from the center pixel$(i,j)$ far away,
that is, at$|a|>\Delta$ or∣$|b|>When\; \Delta$ , set$[\mathbf{V}{]}_{a,b}=0$，最终，我们得到：
$$[\mathbf{H}{]}_{i,j}=u+\sum _{a=-D}\sum _{b=-D}[\mathbf{V}{]}_{a,b}[\mathbf{X}{]}_{i+a,j+b}.$$

This is the convolutional layer (Convolutional Layer) , $V$ is the legendary convolution kernel (Kernel) or filter (Filter),$The\; size\; of\; V$ is$D\times D$

References

6.1. From fully connected layers to convolutions — hands-on deep learning 2.0.0-beta1 documentation