Convolutional neural network-non-linear activation

Convolutional neural network-non-linear activation

First, let's review the classic LeNet network structure mentioned in the article "Introduction to Convolutional Neural Networks" , as shown in Figure 1:

Figure 1 LeNet network structure

We noticed that each convolution operation "Convolution" is followed by a "ReLU" operation. That's right, ReLU here is a non-linear activation function used to perform additional operations after each convolution operation.

ReLU (Rectified Linear Unit), also known as modified linear unit, is a nonlinear operation. The output is as follows:

                                                                                               

Figure 2 ReLU

 ReLU is an element-wise operation (applied to each pixel) and replaces all negative pixel values ​​in the feature map with zero. The purpose of ReLU is to introduce non-linearity in our CNN, because we hope that most of the real-world data that ConvNet learns is non-linear (convolution is a linear operation-element-wise matrix multiplication and addition operations), so we introduce such Non-linear functions such as ReLU to solve non-linear problems.

Tips: "Most real-world data is non-linear", how do you understand this sentence? For example: If you are in the cold winter, holding a torch in your hand can increase your body temperature by 1 degree, then holding 2 torches may increase your body temperature by 1.5 degrees. Because the number of torches and the rise in body temperature are not linear.

Then let's take a look at the modified feature map of the feature map obtained by the convolution operation after the non-linear activation of the ReLU function, as shown in Figure 3:

Question: A careful friend may find that the pixel value range of the image is not 0~255, why is there a negative value in the feature map? (The answer is at the end of the article)

Figure 3 ReLU correction feature map

 Of course, we can also use other nonlinear functions (such as tanh or Sigmoid) instead of ReLU, but it has been found that ReLU performs better in most cases. However, ReLU itself has some drawbacks. For example, negative pixels in the feature map may also contain valuable information, and ReLU sets all of them to zero. Therefore, people have also improved ReLU, with ReLU, Leaky ReLU, PReLU, RReLU, etc.

Answer: It is true that the pixel value of the original image can only be 0~255, but the value in the convolution kernel can be negative, so the feature map obtained by convolution using the convolution kernel will also have negative pixel values.