The principle of BN/LN/GN of neural network regularization

1. Principle of BN layer

torch.nn.BatchNorm2d(num_features, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True,device=None,dtype=None)

• Why use BN?
  Accelerate training. The previous training was slow because during the training process, the overall distribution gradually approached the upper and lower limits of the value range of the nonlinear function (refer to the sigmoid function, large positive or negative values), and the chain derivation leads to the gradient of the neural network in the lower layer disappear. BN is to forcibly pull the increasingly biased distribution back to the (standard) normal distribution, so that the activation value falls in the area where the nonlinear function is more sensitive to the input, so that a small change in the input will cause a large change in the loss function, making the gradient Larger to avoid the vanishing gradient problem.

• Why should the distribution of BN be multiplied by the learnable parameter $c$和$\beta$ ?
  If it is forced to normalize to the standard normal distribution, the distribution learned by this layer before will also lose information. These two reconstruction parameters are introduced so that our network can learn torestorethe feature distribution to be learned by the original network.

• Form(torch) :
  $$y=\frac{x-And[x}{\sqrt{r\left[x\right]\_+ϵ}}\times c+b$$
  

• Backpropagation .
  

• The mean and variance are calculated separately on each dimension of the mini-batch

• Because it is re- CCBN done on$C$$(N,H,W)$ slices, which is tocalculatethe mean and variance of (N, H, W), so it is academically called Spatial Batch Normalization.

• $c$和$\beta$ islearnableparameter whose size is equal to input size. Default$\gamma$ is set to 1,$\beta$ is set to 0.

• The standard deviation is an estimate, the torch.var(input, unbiased=False)same as. $\frac{1}{m}$

• During training, run an estimator that computes the mean and variance, and then normalize during validation.

• Inference phase :

  • $c$ ,$\beta$ directly uses the trained .
  • Use the unbiased estimation of mean and variance . That is, count the mean and variance of each dimension of each batch in training, and then calculate the expectation on the training set:
    $$E\left[x\right]\leftarrow E_{\mathcal{B}}[m_B]$$
    $$r\left[x\right]\_\leftarrow \frac{m}{m-1}{E}_B\left[p_{\mathcal{B}}^2\right]$$
    Finally (just replace the mean and variance, derivation is a small step):
    $$y=\frac{}{\sqrt{r\left[x\right]\_+ϵ}}x+(b-\frac{\gamma E[x]}{\sqrt{r\left[x\right]\_+ϵ}})$$

• Summary of BN advantages :

  • 1. Greatly improve the training speed and speed up the convergence
  • 2. To improve the generalization ability of the network, the explanation is a regular expression method similar to dropout to prevent overfitting, and dropout can be discarded
  • 3. It is easy to adjust parameters, and the requirements for initialization are not so high, and the learning rate can be increased
  • 4. Can disrupt the order of sample training. Can improve accuracy
  • 5. BN is essentially a normalized network layer that can replace the local response normalization layer (LRN) layer

• Why is the BN layer generally used after the linear layer and the convolutional layer instead of after the nonlinear unit ? Because the output distribution shape
  of the nonlinear unit will change during the training process , normalization cannot eliminate its variance offset . On the contrary, the output of the fully connected and convolutional layer is generally a symmetric, non-sparse distribution, which is more similar to Gaussian distribution, normalizing them will produce a more stable distribution . In fact, think about it. For an activation function like relu, if the data you input is a Gaussian distribution, what shape can the data transformed by him be? Those less than 0 are suppressed, that is, the part of the distribution less than 0 directly becomes 0, which is not very high.

• Disadvantages :
  A sufficiently large batch size is required, and a small batch size will lead to an increase in the inaccuracy of the batch statistics and significantly increase the model error rate. That is, BN is greatly affected by the batch. For example, detection and segmentation tasks.

2. LayerNorm principle (LN)

torch.nn.LayerNorm(normalized_shape, eps=1e-05, elementwise_affine=True, device=None, dtype=None)

• Official :
  

• The calculation of mean and variance is similar to BN, but the dimension is normalized_shapedetermined by, for example (3,5), in the last two dimensions.
  

• An example of an image, normalized on C,H,W
  

In CV, along $(C,H,W)$ for normalization.

3. Instance Normlization (IN)和 Group Normalization (GN)

• IN: Adherence $(H,W)$ axis calculation, each sample is calculated separately, and each channel is calculated separately
• GN: Calculate the channel grouping, C/G is the number of channels in each group, along $(X/G,H,W)$ to calculate
  • When G=1, it becomes LN. GN is less restrictive than LN because each group of channels (rather than all channels) is assumed to be affected by a shared mean and variance; the model still has the flexibility to learn a different distribution for each group. This leads to an increase in the representative power of GN relative to LN.
  • When G=C, it becomes GN. But IN can only rely on the spatial dimension to calculate the mean and variance, and misses the opportunity to exploit channel dependence.

4. Summary

• BN: Normalize in the batch direction, calculate $N\ast H\ast$mean of$W$
• LN: Normalize in the channel direction, calculate $C\ast H\ast$mean of$W$
• IN: Normalize in a channel, calculate $H\ast$mean of$W$
• GN: First divide the channel direction into groups, and then perform normalization in each group, and calculate $(C//G)\ast H\ast$mean of$W$