Is Huawei starting to make big moves again? CV New Architecture: VanillaNet: the Power of Minimalism in Deep Learning Paper Reading Notes

write in front

  At the VALSE conference two days ago, Huawei's article shocked me. To be honest, the performance was a bit explosive. After looking at the table on Github, the number of parameters is scary as the number of layers increases, but the inference speed is much higher than the previous models.

• Paper address: VanillaNet: the Power of Minimalism in Deep Learning
• Code address: https://github.com/huawei-noah/VanillaNet
• Pre-training model address: https://gitee.com/mindspore/models/tree/master/research/cv/vanillanet
• Estimated submission: a top conference in 2024
• Interpretation of other WeChat articles: Huawei's Noah minimalist network achieves 83% accuracy with 13 layers (source code attached)
• Ps: A blog post reading notes every week for 23 years, and more useful information on the homepage . Welcome to follow me. I look forward to your participation in the 5,000 fans~

1. Abstract

  Looking at the entire Abstract, there is no specific content, that is to say, the VanillaNet proposed in this article can use ResNet with a left hook, Swim-Transformer with a right hook, etc. The main reason is to avoid the introduction of ultra-deep, shortcuts, and self-attention mechanisms, and there are no complex activation functions.

2. Introduction

  The first paragraph briefly talks about the development and role of AI, and the second paragraph starts with the introduction of AlexNet and the follow-up of ResNet, which shows that this model is becoming more and more complex in design, but at the same time its performance is getting better and better. The third section is the home of Transformer, still emphasizing the depth of the model. The fourth paragraph is a transition from previous to next, indicating that deeper and more complex model structures are difficult to deploy. The fifth paragraph shows the problem. Flat networks will have gradient disappearance problems, and the performance of some deep networks far exceeds the previous AlexNet and VGG, so few people pay attention to the design of the model structure.
  The sixth paragraph introduces the topic. This article proposes VanillaNet. The model structure is simple: the ultra-depth, shortcut branches, and self-attention operations of the model are removed. Accordingly, a training strategy is proposed to gradually eliminate nonlinear layers to maintain the inference speed. In order to enhance the nonlinearity of the network, a series-based activation function is proposed, which is far more effective than other models. One last thing, VanillaNet is so powerful, come and follow my work.

3. Neural structure of a single vanilla

  Most SOTA classification models consist of three parts: the stem block turns the input 3-channel image into multi-channel and is accompanied by downsampling, the main body module is used to learn useful information, and a fully connected layer is used for the classifier. output. The body module has four stages, each stage is stacked by multiple identical blocks. After each stage the number of feature channels will increase, while the width and height will decrease.
  In the next paragraph, I complain that ResNet and ViT are too deep and too deep, and ViT requires multiple self-attention layers.
  In the current development of AI chips, the FLOPs or parameter quantities that were originally restricted are no longer the bottleneck, because Lao Huang's NVIDIA GPU has indeed developed. Therefore, the complex design of the model and deeper blocks have become the main factors restricting the speed. So this article proposes VanillaNet, as shown in the figure below:

VanillaNet is still a three-stage design, but the difference is the depth. Each stage is built with only one layer.

  Take the 6-layer VanillaNet as an example, stem: convolutional layer $4\times 4\times 3\times C$ , step size is$4$ , will$The\; input\; image\; of\; the\; 3-$ layer channel is mapped toCCon channel$C.$On stages 1, 2, and 3, use a step size of2 2A max pooling layer of$2$$2$ times. In stage 4, the number of channels is not increased due to subsequent average pooling. The last fully connected layer outputs the classification result. The kernel size of each convolutional layer is$1\times 1$ , followed by an activation layer and a layer of Batch normalization. This is the entire structure, no shortcuts or extra blocks.
  Since VanillaNet is relatively simple and a shallow network, which weakens the performance of the model, a series of techniques are proposed to enhance its nonlinearity.

4. Training VanillaNet

4.1 Deep training strategy

  The strategy of training two convolutional layers plus an activation layer is different from training a single convolutional layer. It requires gradually reducing the activation function as the training epoch increases. At the end of training, the two convolutional layers can be fused into one to reduce inference time.
  For an activation function $A\left(x\right)$ , such as ReLU and Tanh, are combined with a unique mapping:
$$A^{\prime }(x)=(1-l)A\left(x\right)+\lambda x$$
whereλ$\lambda$ is the equilibrium modified$A^{\prime }\left(x\right)$nonlinear hyperparameters of the activation function. order$and$、$E$ represents the current epoch and the overall number of epochs respectively, then$l=\frac{e}{E}$. So at the beginning of training, $e=0,A^{\prime }(x)=A\left(x\right)$ means that the model has strong nonlinearity. As the training converges, we finally have$A^{\prime }\left(x\right)=x$ , indicating that there is no activation function between the two convolutional layers.

  Next, the batch normalization and subsequent convolution of each layer are turned into a single convolution operation.

令 $W\in {\mathbb{R}}^{C_{out}\times (C_{in}\times k\times k)}$，$B\in {\mathbb{R}}^{C_{out}}$are the weights and biases of the convolutional layer respectively, input $C_{in}$channels, output $C_{out}$channels, the convolution kernel size is $k$ . The scale, translation, average and differential of batch normalization are respectively$c,b,m,p\in R^{C_{out}}$Display, you can display the fusion operation of the batch version:
$$W_i^{\prime }=\frac{c_i}{p_i}{W}_i,B_i^{\prime }=\frac{(B_i-m_i)c_i}{p_i}+b_i$$where subscript i ∈ { 1 , 2 , … , C out } i\in\{1,2,\ldots,C_{out}\}i∈{ 1,2,…,Cout​} represents the$The\; output\; value\; of\; i$ channel.

  Then merge the two $1\times 1$ convolution. The input and output features are respectively expressed as$x\in {\mathbb{R}}^{C_{in}\times H\times W}$、$y\in R^{C_{out}\times H^{\prime }\times W^{\prime }}$ , so the convolution operation can be expressed as:
$$y=W\ast x=W\cdot \mathrm{im}2\mathrm{col}(x)=W\cdot X$$ where$\ast$ represents the convolution operation,$\cdot$ represents matrix multiplication,$X\in {\mathbb{R}}^{(C_{in}\times 1\times 1)(H^{\prime }\times W^{\prime })}$originates from$\text{The im2col}$ operation converts the input into a shape corresponding to the convolution kernel. For$1\times For\; convolution$ , there is no need to slide the convolution kernel on the overlapping part (because there is no overlapping part). So the weight matrix of the two convolutional layers is expressed as$W^1$ sum$W^2$ , two convolution operations without activation function can be expressed as:
$$y=W^1\ast (W^2\ast x)=W^1\cdot W^2\cdot \text{im}2\text{col}(x)=(W^1\cdot W^2)\ast X$$ So far, two$1\times 1Convolutions$ are smoothly fused and do not reduce the inference speed.

4.2 Series Informed Activation Function

  Some of the current mainstream activation functions include Rectified Linear Unit (ReLU) and its variants PReLU, GeLU, and Swish, which are limited by the nonlinearity of simple and shallow networks. Compared with deep networks, these activation functions have not been systematically studied. .
  There are two ways to improve the nonlinearity of neural networks: stacking nonlinear activation layers or increasing the nonlinearity of each activation layer. Most mainstream networks choose the former, which leaves high potential for parallel computing.
  One of the direct ideas is to improve the nonlinear ability of stacked activation layers. Stacking activation layers in series is the key to deep networks. In contrast, turning to stacking activation layers simultaneously is a good approach. Denote the single-layer activation function as $A\left(x\right)$ ,$x$ is the input, and the function can be ReLU and Tanh. Stack$A\left(x\right)$ possible display:
$$A_s(x)=\sum _{i=1}^n{a}_{i}A(x+b_i)$$ where$n$ represents the number of stacked activation functions,$a_i$、$b_i$are the scale and bias of the activation function respectively. The non-linearity of the activation function can be enhanced by simultaneous stacking.
  In order to further enhance the approximation ability of Series, the input is changed by changing its neighbors to learn global information, similar to BNET. Specifically, given the input feature $X\in {\mathbb{R}}^{H\times W\times C}$ , where$H$、$W$、$C$ are the input width, height, and number of channels respectively. So the activation function is shaped as follows:
A s ( xh , w , c ) = ∑ i , j ∈ { − n , n } ai , j , c A ( xi + h , j + w , c + bc ) A_s(x_{ h,w,c})=\sum\limits_{i,j\in\{-n,n\}}a_{i,j,c}A(x_{i+h,j+w,c}+ b_c)As​(xh,w,c​)=i,j∈{ −n,n}∑​ai,j,c​A(xi+h,j+w,c​+bc​) whereh ∈ { 1 , 2 , … , H } h\in\{1,2,\ldots,H\}h∈{ 1,2,…,H}、$w\in \{1,2,\dots ,W\}$、$c\in \{1,2,\dots ,C\}$ . When$n=0$ 时，$A_s(x)=A\left(x\right)$ . This article uses ReLU as the activation function to build the series.
  Next, analyze its computational complexity: for a convolution kernel with size$The\; convolutional\; layer\; of\; K$ has the input and output channel numbers C in C_{in}respectively.$C_{in}$、$C_{out}$, the computational complexity is:
$$\mathcal{O}(\mathrm{CONV})=H\times W\times C_{in}\times C_{out}\times k^{2The}$$ cost of the series activation layer is:
$$O\left(\text{SA}\right)=H\times W\times C_{in}\times n^2$$于是：
$$\frac{O(\text{CONW}}{O\left(\text{SA}\right)}=\frac{H\times W\times C_{in}\times C_{out}\times K^2}{H\times W\times C_{in}\times n^2}=\frac{C_{out}\times k^2}{n^2}$$Ranked 4th in VanillaNet-B $Take\; 4$ stages as an example, where$C_{out}=2048$，$k=1$，$n=7$ , the ratio of the above formula is$84$ . Therefore, the computational complexity of the proposed activation layer is lower than that of the convolutional layer.

5. Experiment

  ImageNet dataset.

5.1 Ablation experiment

The influence of the number of series in the activation function

The impact of training techniques

The impact of shortcut branches

5.2 Visualization of attention

5.3 Comparison with SOTA architecture

5.4 Experiments on COCO dataset

6. Conclusion

  This article studies the feasibility of simple, shallow neural networks, and proposes a deep training strategy and series activation function for training VanillaNets to enhance the nonlinearity of the model. The experimental results show that VanillaNets is very effective, and I hope everyone will give it a try.

Appendix A: Network structure

Each convolutional layer is followed by an activation layer. For VanillaNet-13-1.5×, the number of channels x1.5. For VanillaNet-13-1.5׆, adaptive pooling is further used in stages 2, 3, and 4, and the corresponding shape is $40\times 40$、$20\times 20$、$10\times 10$。

Appendix B: Training details

write on the back

  This Huawei article is short and concise, and it really reflects the basic skills. In this era of "deep" learning, it is truly awesome to dare to challenge shallow networks.