Wu Enda Convolutional Neural Network Chapter Notes (1) - Convolutional Neural Network

Video course link:
https://www.bilibili.com/video/BV1FT4y1E74V?
Note reference link:
https://blog.csdn.net/weixin_36815313/article/details/105728919

1. Edge Detection Example

1.1 Convolution operation

Convolution operation is the most basic component of convolutional neural network.

Suppose there is a $6$×$6$ grayscale images. Because it is a grayscale image, that is, there is no RGB three-channel, so it is$6$×$6$×$1$ matrix (as shown above). To detect vertical edges in an image, you can construct a$3$×$3$ matrix. In the terminology of convolutional neural networks, it is calleda filter (filter), or also calleda kernel (kernel).

Assume constructing a $3$×$3$ for the filter (as shown above), 6for the above$6$×$The\; image\; of\; 6$ is convoluted, and the convolution operation uses$\ast$ to represent.

The output of this convolution operation will be a $4$×$4$ matrix, you can think of it as a$4$×$4$ image (as above).

first calculate $4$×$4$ The first element in the matrix, use$3$×$3$ filter, cover it on the input image, and then perform element-wise products (element-wise products) operation, namely$\left[\begin{array}{ccc}3\times 1& 0\times 0& 1\times (-1)\\ 1\times 1& 5\times 0& 8\times (-1)\\ 2\times 1& 7\times 0& 2\times (-1)\end{array}\right]=\left[\begin{array}{ccc}3& 0& -1\\ 1& 0& -8\\ 2& 0& -2\end{array}\right]$，最后将该矩阵每个元素相加得到最左上角的元素，即$3+1+2+0+0+0+(-1)+(-8)+(-2)=-5$。

接下来计算第二个元素，因此要把$3$×$3$的过滤器向右移动一步（如上图）。继续做同样的元素乘法，然后加起来，即$0\times 1+5\times 1+7\times 1+1\times 0+8\times 0+2\times 0+2\times (-1)+9\times (-1)+5\times (-1)=-4$。然后继续将$3$×$3$的过滤器右移，直到计算完这一行的元素。

接下来为了计算下一行的元素，将$3$×$3$的过滤器下移（如上图）。重复进行元素乘法，然后加起来。通过计算得到$-10$ . By analogy, other elements in the matrix are calculated.

Therefore $6$×$6$ matrix and$3$×$3$ matrices areconvolvedto get$4$×$4$ matrix. These pictures and filters are matrices of different dimensions, but the matrix on the left is easily understood as a picture, the one in the middle is understood as a filter, and the picture on the right can be understood as another picture. This isthe vertical edge detector.

1.2 Application Case of Vertical Edge Detector

The picture above is a simple $6$×$6$ image, the left half of the pixel value is 10, and the right half of the pixel value is 0. If you think of it as a picture, the part with a pixel value of 10 on the left is brighter, and the part with a pixel value of 0 on the right is darker. There is a particularly noticeable vertical edge in the middle of the picture, and this vertical line is the transition line from black to white.
When you use$3$×$3$ filters perform convolution operations, this$3$× 3 3 The filter of$3\; can\; be\; regarded\; as\; such\; a\; picture,\; the\; left\; side\; is\; brighter,\; there\; is\; a\; transition\; in\; the\; middle,\; the\; color\; is\; darker\; than\; the\; left\; side,\; and\; then\; the\; right\; side\; is\; darker.$

After the convolution operation, the matrix on the right is obtained. If you put this $4$×$4$ The matrix is ​​regarded as an image, and it can be seen that there is a brighter area in the middle, and darker areas on both sides, indicating that there is a particularly obvious vertical edge in the middle of the image.

2. More Edge Detection Content (More Edge Detection Example)

in computer vision, $3$×$3$ Several conventional digital combinations of filters:

• Vertical edge filter: $$\left[\begin{array}{ccc}1& 0& -1\\ 1& 0& -1\\ 1& 0& -1\end{array}\right]$$
• Horizontal edge filter: $$\left[\begin{array}{ccc}1& 1& 1\\ 0& 0& 0\\ -1& -1& -1\end{array}\right]$$
• Sobel filter: Its advantage is that it increases the weight of the elements in the middle row, which makes the result more robust. $$\left[\begin{array}{ccc}1& 0& -1\\ 2& 0& -2\\ 1& 0& -1\end{array}\right]$$
• Scharr equation: $$\left[\begin{array}{ccc}3& 0& -3\\ 10& 0& -10\\ 3& 0& -3\end{array}\right]$$

In deep learning, all we need to do is to treat the 9 numbers in this matrix as 9 parameters, and use the backpropagation algorithm to let the neural network learn whatever it needs 3 $3$×$3$ filter, and apply it on the whole image, the goal is to understand these 9 parameters.

3. Padding

In order to build a deep neural network, one of the basic convolution operations you need to learn to use is padding .
In the example in the previous chapter, if a $3$×$3$ filters convolved with a$6$×$6$ images, will end up with a$4$×$4$ output, which is a$4$×$4$ matrix. That's because$3$×$3$ filters at$6$×$In\; the\; matrix\; of\; 6$ , there are only 4×4 possible positions. The mathematical explanation behind this is that if we have a$n$×$image\; of\; n$ , with$f$×$The\; filter\; of\; f$ is convolved, then the dimension of the output image is$(n-f+1)$×$(n-f+1)$ .
However, there are two disadvantages in this process:
1. Disadvantage 1:Output shrinkage

• Every time a convolution operation is performed, the output image will be reduced, such as from 6 6 in the previous example$6$×$6$ shrinks to$4$×$4$ . If there is a deep neural network, if the image is reduced every time it passes through a layer, a very small picture will be obtained.

2. Disadvantage 2: Information loss at the edge of the image

• from $6$×$As\; seen\; in\; the\; image\; of\; 6$ , the pixels at the corner edges (marked with green shading) are only$3$×$3$ 's filter is convolved once, i.e. used only once in the output, since it is located in this$3$×$3$ corner of the area. But if it is in the middle pixel (marked by the red box), there will be many$3$×$3$ 's area overlaps with it. So those pixels in the corner or edge area are used less in the output, which also means that a lot of information about the edge position of the image is lost.
  

In order to solve the above two problems, there are usually the following solutions:

• Padding the image before the convolution operation . You can pad another layer of pixels along the edge of the image, then $6$×$The\; image\; of\; 6$ is filled into an$8$×$8$ images. At this time, if you use$3$×$3$ filters on this$8$×$8$ image convolution, you get a$6$×$6$ images. In general, it can be filled with pixel value 0.
  

Because we have filled a pixel around, the number of filling $p=1$ , the size of the output becomes$(n+2p-f+1)$×$(n+2p-f+1)$ . In this way, the disadvantage that information in the corners or edges of the image plays a lesser role is mitigated.
In fact, more pixels can be filled. As for how many pixels to choose to fill, there are usually two options, calledValid convolutionandSame convolution.

• Valid convolution - no padding. if there is a $n$×$image\; of\; n$ , with a$f$× f f The filter convolution of$f$$(n-f+1)$×$(n-f+1)$ dimensional output.
• Same convolution - After padding, the output size is the same as the input size. if you have a $n$× n n image of$n$$p$ pixels fill the edge, the size of the output is$(n+2p-f+1)$×$(n+2p-f+1)$ . So if you want$n+2p-f+1=In\; the\; case\; of\; n$ , the output and input are equal in size, then$p=(f-1)/2$。

Traditionally, in computer vision, $f$ is usually an odd number. There are several possible reasons for this:

1. if $f$ is an even number, then you can only use some asymmetric padding. onlyffWhen$f$ is an odd number, Same convolutionwill have natural padding. We can fill the surroundings with the same amount, instead of filling more on the left and less on the right, so that the filling is asymmetrical.
2. When you have a filter with an odd number of dimensions, say $3$×$3$ or$5$×$5$ , it has a center point. Sometimes in computer vision, it is easier to point out the position of the filter if there is a central pixel.

4. Strided Convolutions

The stride in convolution is another basic operation in building convolutional neural networks.

Suppose using a $3$×$3$ filters to convolve this$7$×$7$ image, take the upper left 3 3as before$3$×$The\; products\; of\; the\; elements\; in\; the\; 3$ regions are summed again, and the final result is 91.

The difference from before is that we set the stride to 2. So next put $3$× 3 3 A filter of$3\; moves\; two\; steps\; to\; the\; right,\; multiplies\; and\; sums\; each\; element,\; and\; gives\; a\; result\; of\; 100.$

Then proceed to $3$× 3 3 The filter of$3\; is\; moved\; by\; two\; steps,\; and\; the\; result\; of\; calculation\; is\; 83.$

After completing the first line, move to the next line, and at this time also move two steps (as shown in the figure above). Similarly, multiply and sum the elements, resulting in 69.

And so on, finally get a $3$×$3$ output (as shown above). If using a$f$× f f filter convolution of$f$$n$×$n$ image,paddingis$p$ witha strideof$s$ , then the size of the output is$\lfloor \frac{n+2p-f}{s}+1\times \frac{n+2p-f}{s}+1\rfloor$ .
Note: Symbols$\lfloor \rfloor$ meansround down, that is, if the calculated quotient is not an integer, it will be rounded down.

5. Convolutions Over Volumes

The previous chapter talked about convolution of two-dimensional images, and this chapter is about convolution of three-dimensional stereo .

5.1 Calculation of 3D convolution

Suppose there is a $6$×$6$×$3$ RGB color images, here$3$ refers to three color channels. To detect edges or other features of an image, use a$3$×$3$×$3$ filters are convolved, and this filter also has three layers, corresponding to the three channels of red, green, and blue.

To compute the output of this convolution operation, all you have to do is take this $3$×$3$× 3 3 The filter of$3$$3$×$3$×$The\; filter\; of\; 3$ has 27 numbers, take these 27 numbers in turn, multiply by the number of the corresponding channel in the input image, that is, firsttake the first 9 numbers of the red channel, multiply by the 9 numbers of the corresponding red channel in the input image, Then take the 9 numbers of the green channel, multiply by the 9 numbers of the corresponding green channel in the input image, and finally take the 9 numbers of the blue channel, multiply by the 9 numbers of the corresponding green channel in the input image, and multiply by Input the 9 numbers corresponding to the blue channel in the image, and then add these numbersto get the first number of the output.

Then slide the cube by one unit, then multiply and add the 27 numbers with the numbers in the 3 channels corresponding to it, and then get the next output, and so on.

5.2 3D Convolution with Multiple Filters

In actual image processing, it is difficult to fully extract complex image features only by a single filter. So in general, we use far more than one filter for convolution.

We convolve with the first filter to get a $4$×$4$ output, and then convolved with the second filter to get a 4×4 output, put the second output behind the first output, thus forming a 4×4×2 output, here 2 represents the number of filters.
In general, if you have a$n$×$n$×$n_c$The input image, convolution on a $f$×$f$×$n_c$filter, and then you get a $(n-f+1)$×$(n-f+1)$×$n_{c^{{}^{\prime }}}$output image, here $n_{c^{{}^{\prime }}}$Indicates the number of filters and also the number of channels in the next layer.
Note: Here it is assumed that the stride is 1 and there is no padding .

6. One Layer of a Convolutional Network

6.1 A simple example of a single-layer convolutional network

For the example in the previous chapter, a three-dimensional image is processed using two filter convolutions, outputting two different 4×4 matrices. First add bias b 1 b_1 to the first output$b_1$(here $b_1$is a real number), and then apply a nonlinear activation function (reLu function is used here), the output is a $4$×$4$ matrix.
For the second output add a different bias$b_2$(here $b_2$is a real number), then apply a non-linear activation function (reLu function is used here), and finally get another $4$×$4$ matrix. These two matrices are then stacked to end up with a$4$×$4$×$2$ matrix. The forward propagationprocess
in a single-layer convolutional neural network(1)z [ l ] = w [ l ] a [ l − 1 ] + b [ l ] z^{[l]}=w^{[l] }a^{[l-1]}+b^{[l]}
$z^{[l]}=w^{[l]}{a}^{[l-1]}+b^{[l]}$
(2) $a^{[l]}=g(z^{\left[l\right]})$
Among them$w^{[l]}{a}^{[l-1]}$ corresponds to$6$×$6$×$The\; input\; image\; of\; 3$ and$3$×$3$×$3$ filters convolved to get$4$× 4 4 The output matrix of$4$$z^{\left[l\right]}$ corresponds to the added offset of$4$×$4$ output matrix (as shown below).

This is $a^{\left[l\right]}$到$a^{}$The evolution process of${}^{[}$${}^l$${}^{+}$${}^1$${}^{]\; first\; executes\; the\; linear\; function,\; and\; then\; multiplies\; all\; elements\; to\; perform\; convolution.\; The\; specific\; method\; is\; to\; use\; the\; linear\; function\; plus\; bias,\; and\; then\; apply\; the\; activation\; function\; ReLU.}$In this way, a$6$×$6$×$3$ of dimension$a^{\left[0\right]}$ evolves into a$4$×$4$×$2-$ dimensional$a^{\left[1\right]}$ , this is a layer of a convolutional neural network.
Suppose each layer is$3$×$3$×$3$ filters, so each filter has 27 parameters, plus a bias$b$ , a total of 28 parameters. Now we have 10$3$×$3$×$3$ filters add up to 28×10, which is 280 parameters.
No matter how big the input image is, the number of parameters is always 280. Use these 10 filters to extract features such as vertical edges, horizontal edges and other features. Even though these images are large, there are very few parameters, which is a feature of convolutional neural networks calledavoiding overfitting.

6.2 Symbol Definition

Take the llth in the convolutional neural network$Take\; the\; l-$ layer convolutional layer as an example to define various marks of the convolutional layer.

• $f^{\left[l\right]}$ means the$The\; size\; of\; the\; l-$ layer filter;
• $p^{\left[l\right]}$ means the$The$ number of layerspadding,the number of paddingcan be specified asvalid convolution, that is, nopadding, orthe same convolutionpaddingis selected;
• $s^{\left[l\right]}$ means thellthThe strideof layer$l$ ;
• No. llThe dimension of the input image of layer$l$$n_H^{[l-1]}$×$n_W^{[l-1]}$× $n_c^{[l-1]}$, which is the activation value of the previous layer;
• No. $The\; dimension\; of\; the\; output\; image\; of\; layer\; l$ is$n_H^{[l]}$×$n_W^{\left[l\right]}$× $n_c^{\left[l\right]}$；
• $n_H^{\left[l\right]}$Indicates the llthThe height of the output image of layer$l$$n_H^{\left[l\right]}=\lfloor \frac{n_H^{[l-1]}+2p^{[l]}-f^{\left[l\right]}}{s^{[l]}}+1\rfloor$；
• $n_W^{[l]}$Indicates the llthThe width of the output image of layer$l$$n_W^{\left[l\right]}=\lfloor \frac{n_W^{[l-1]}+2p^{[l]}-f^{\left[l\right]}}{s^{[l]}}+1\rfloor$；
• $n_c^{\left[l\right]}$Indicates the number of channels of the output image.
• The dimension of a single filter is $f^{[l]}$×$f^{\left[l\right]}$ ×$n_c^{[l-1]}$, the number of channels in the filter must be consistent with the number of channels in the input image;
• For a single sample, the activation value $a^{\left[l\right]}$ is a three-dimensional body whose dimension is$n_H^{[l]}$×$n_W^{\left[l\right]}$× $n_c^{\left[l\right]}$; for $m$ samples, ie$m$ activation values$a^{\left[l\right]}$ set, then output$A^{\left[l\right]}$ dimension is$m$×$n_H^{\left[l\right]}$×$n_W^{\left[l\right]}$× $n_c^{\left[l\right]}$；
• Weight parameter $W^{\left[l\right]}$ is the set of all filters multiplied by the total number of filters, its dimension is$f^{[l]}$×$f^{\left[l\right]}$ ×$n_c^{[l-1]}$× $n_c^{[l]}$；
• Bias parameter $b^{\left[l\right]}$ is usually represented in code as a$1$×$1$×$1$ ×$n_c^{\left[l\right]}$A 4D vector or 4D tensor of . Each filter has a bias parameter, which is a real number.
  

7. A Simple Convolution Network Example

Suppose you input a picture, defined as $x$ , for image classification or image recognition. The size of the input image is$39$×$39$×$3$ , because$n_H^{[0]}=n_W^{\left[0\right]}$, that is, both height and width are equal to 39, $n_c^{\left[0\right]}=3$ , that is, the channel number of layer 0 is 3.
Suppose we use 10$3$×$3$×$3$ filters to extract features, that is,filter size$f^{[1]}=3$ ,the number of filters$n_c^{[1]}=10$ .$\_$The height and width usevalid convolution, that is,the number of padding$p^{\left[1\right]}=0$ ,stride$s^{\left[1\right]}=1$ .
Therefore, the size of the output image obtained by convolution, or the activation value of the first layer$a^{\left[1\right]}$ has dimension$37$×$37$×$10$ , where 37 is the formula$\frac{n+2p-f}{s}+The\; calculation\; result\; of\; 1$ , that is,$\frac{39+0-3}{1}+1=37$ . This completes the construction of the first convolutional layer.

Then go to the next convolutional layer, this time we use a filter of $5$×$Matrix\; of\; 5$ ,filter size$f^{[2]}=5$ , because the number of channels of the output image of the previous layer$n_c^{[1]}$is 10, so the filter dimension of this layer is $5$×$5$×$10$ .$\_$In addition,the stride$s^{[2]}=2$ ,padding amount$p^{\left[2\right]}=0$ , and there are 20 such filters, so the dimension of the output of this layer is$17$×$17$×$20$ , namelythe height and width of the output image$n_H^{[2]}=n_W^{\left[2\right]}=17$ ,number of channels$n_c^{\left[2\right]}=20$ .$\_$Because the stride is 2, the dimension shrinks quickly, so the dimension size goes from$37$×$37$ reduced to$17$×$17$ , reduced by more than half. This completes the construction of the second convolutional layer.

Then build the last convolutional layer, assuming the filter is still $5$×$5$ , iefilter size$f^{[3]}=5$ , because the dimension of the input image is$17$×$17$×$20$ , so the filter dimension of this layer is$5$×$5$×$20$ .$\_$Supposestride$s^{\left[3\right]}=2$ ,padding amount$p^{\left[3\right]}=0$ , and 40 such filters are used, so the final output dimension is$7$×$7$×$40$ .$\_$Here, this$39$×$39$×$The\; input\; image\; of\; 3$ is processed, and$7$×$7$×$40\; features,\; that\; is,$ 1960 features.

Then perform convolution processing on these 1960 features, which can be smoothed or expanded into 1960 units, which can be saved into a long vector after smoothing, and then input to the logistic regression unit or softmax regression unit , and finally use $\hat{y}$Represents the predicted output of the neural network.

8. Pooling Layers

In addition to convolutional layers, convolutional networks often use pooling layers to reduce the size of the model, increase computational speed, and improve the robustness of the extracted features.

8.1 Max Pooling

If the input is a $4$×$4$ matrix, the tree pooling that performs max pooling is a$2$×$2$ matrix, ie$f=2$ . In order to calculate the values ​​of the 4 elements on the right, we need$2$×$2$ area to do the maximum value calculation. First on purple$2$×$The\; 2$ area calculates the maximum value and the result is 9. Suppose stride$s=2$ , so move 2 more strides to the right to get the blue$2$×$The\; maximum\; value\; in\; the\; 2$ area is 2. Then move down 2 steps to get the green$2$×$The\; maximum\; value\; in\; the\; 2$ area is 6. Finally move 2 more steps to the right to get the red$2$×$The\; maximum\; value\; in\; the\; 2$ area is 3. The final output gets a$2$×$2$ matrices.
for this$2$×$2$ , each element of the output is the maximum element value in its corresponding color area. wherefilter size$f=2$ andstride$s=2$ is the hyperparameter of max pooling, but these two hyperparameters are fixed values ​​because they cannot be learned in gradient descent.

8.2 Average Pooling

As the name suggests, this operation does not select the maximum value of each filter, but the average value. Assume filter size $f=2$ , stride$s=2$ , so the average value of the purple area is 3.75, the average value of the blue area is 1.25, the average value of the green area is 4, and the average value of the red area is 2.
Currently, max pooling is more commonly used than average pooling. But there are exceptions, that is, deep neural networks, you can use average pooling to decompose the size of$7$×$7$×$The\; output\; matrix\; of\; 1000$ , averaged over the whole space, finally got$1$×$1$×$1000$ output matrix.

9. Convolutional neural network example

Suppose there is a sheet of size $32$×$32$×$3$ picture, thisRGBpicture contains a handwritten number, such as 7, you want to identify which of the 10 numbers it is 0-9, so let's build a convolutional neural network to achieve this function.

First build the first layer of convolutional layer, the input is a $32$×$32$×$3$ matrix, assuming that the dimension of the filter used in the first layer is$5$×$5$×$3$ ,the strideis 1,the paddingis 0, andthe number of filtersis 6, then the dimension of the output matrix of the convolutional layer is$28$×$28$×$6$ , label this layerCONV1.
Then build a pooling layer immediately after the convolutional layer, here choose to usethe largest pooling. Suppose the parameter$f=2$，$s=2$，$p=0$ , that is, the filter used by the maximum pooling is$2$×$2$ ,the strideis 2,paddingis 0, so the height and width of the output will be reduced by half, the number of channels remains the same, and the final output matrix dimension is$14$×$14$×$6$ , mark the pooling layer asPOOL1.
When people calculate the number of layers of a neural network, they usually only count the layers with weights and parameters, while the pooling layer has no weights and parameters, only some hyperparameters. So here we takeCONV1andPOOL1together as a convolution and mark it asLayer1.
Note: Another type of division method is to usethe convolutional layeras a layer, andthe pooling layeras a separate layer.

Next apply another convolutional layer to Layer1 , assuming the dimension of the filter is $5$×$5$×$3$ , ie$f=5$ ,the strideis 1,the paddingis 0, andthe number of filtersis 16, so the output dimension of the convolutional layerCONV2is$10$×$10$×$16$ .$\_$
Continue to perform the maximum pooling calculation, assuming that the parameter is$f=2$，$s=2$ , so the height and width are halved, the number of channels is the same as before, and the dimension of the output matrix is$5$×$5$×$16,$ labeledPOOL2. The convolutional layerCONV2and the pooling layerPOOL2are a convolution, that is,Layer2, because it has only one weight set and one convolutional layerCONV2.

The output of Layer2 is a $5$×$5$×$A\; matrix\; of\; 16$ contains 400 elements. NowPOOL2into a 1D vector of size 400. We can imagine the flattened result as such a collection of neurons, and then use these 400 units to build the next layer.
The next layer contains 120 units, and these 400 units are closely connected to 120 units, which is our firstfully connected layer. It is equivalent to a single neural network layer, and its weight matrix is$W^{\left[3\right]}$ , dimension is$120$×$400$ , there is also a bias parameter$b^{\left[3\right]}$ , dimension is$120$×$1$ .
Then we add afully connected layer, assuming it contains 84 units, labeledFC4. Finally a softmaxis filled with these 84 units. If we want to recognize 10 numbers from 0-9 through handwritten digit recognition, thissoftmaxunit will have 10 outputs.
Regarding how to select these hyperparameters, the general practice is to try not to set hyperparameters yourself, but to check which hyperparameters have been adopted by others in the literature, and choose an architecture that works well in other people's tasks, then it may also apply to yourself s application.
Another common pattern in neural networks is one or more convolutions followed by a pooling layer, then one or more convolutional layers followed by another pooling layer, then several fully connected layers, and finally Asoftmaxlayer.
Next, let's talk about the activation value dimension, activation value size and number of parameters of the neural network.

There are the following points to note:

• The pooling layer and the max pooling layer have no parameters;
• Convolutional layers have relatively few parameters, and many parameters exist in fully connected layers of neural networks.

It can be seen from the above figure that as the depth of the neural network deepens, the size of the activation value will gradually become smaller. If the size of the activation value decreases too quickly, it will also affect the performance of the neural network. In the example, the activation value size is 6000 in the first layer, then reduced to 1600, slowly reduced to 84, and finally the softmax result is output.

10. Why use convolution? (Why Convolutions?)

The two main advantages of convolutional layers over using only fully connected layers are parameter sharing and sparse connections .

Suppose there is a $32$×$32$×$3$ images, using 6 dimensions for$5$×$5$×$3$ filters, the dimension of the output matrix is$28$×$28$×$6$ . 32×32×3=3072, 28×28×6=4704. Suppose we build an ordinary neural network, one layer contains 32×32×3=3072 units, the next layer contains 28×28×6=4704 units, each neuron in the two layers is connected to each other, and then calculate Weight matrix$W$ , its dimension is equal to 4074×3072≈14 million, so many parameters need to be trained. Although with the current technology, we can use more than 14 million parameters to train the network, because this 32×32×3 picture is very small, there is no problem in training so many parameters. But if this is a$1000$×$1000$ picture, weight matrix$W$ can get very large.
If we build a convolutional layer, the dimension of each filter is$5$×$5$×$3$ , a filter has 75 parameters, plus the bias parameter$b$ , then each filter has 76 parameters, because there are 6 filters in total, so the total number of parameters is 456. It can be seen that relatively speaking, the number of parameters involved in the convolution operation is significantly reduced.

• Parameter sharing — assuming a $3$× 3 3 The filter of 3 detects vertical edges, then the upper left corner of the picture, and each area next to it (the part marked by the blue box in the left matrix) can use this$3$$3$×$3$ filters, that is,each feature detector and output can use the same parameters in different regions of the input imagein order to extract vertical edges or other features. It is not only suitable for low-order features such as edge features, but also for high-order features, such as extracting eyes on faces, cats or other feature objects.
• Sparse connection — eg 0 in the upper left corner of the output matrix is ​​through $3$×$3$ convolution calculation, it only depends on this$3$× 3 3 In the input cell of 3, a certain element of the output matrix on the right is only connected to 9 of the 36 input features, and other pixel values ​​​​do not have any impact on the output$.$

Convolutional neural networks can reduce parameters through these two mechanisms so that we can train it with a smaller training set, thereby preventing overfitting.