Deep learning-BN (Batch Normalization)

1. Introduction
Batch Normalization is proposed in a paper in 2015. $Data\; normalization$ methods are often used before activation layers in deep neural networks. Its role can speed up the convergence speed of model training, make the model training process more stable, and avoid gradient explosion or gradient disappearance. And it plays a certain regularization role, almost replacing Dropout.
2. Formula
I nput : B = { x 1... m } ; λ , β ( parameterstobelearned ) Input:B = \lbrace x_{1...m}\rbrace;\lambda,\beta(parameters\quad to \quad be\quad learned)Input:B={ x1...m​};l ,β(parameterstobelearned) O u t p u t : { y i = B N λ , β ( x i ) } Output:\lbrace{y_i=BN_{\lambda,\beta}(x_i)}\rbrace Output:{ yi​=BNl , b(xi​)} $$m_B\leftarrow \frac{1}{m}\sum _{i=1}^m{x}_i$$ $$p_B^2\leftarrow \frac{1}{m}\sum _{i=1}^m(x_i-m_B{)}^2$$ $$\overline{x_i}\leftarrow \frac{x_i-m_B}{\sqrt{p_B^2+ϵ}}$$ $$y_i\leftarrow c\overline{x_i}+b$$

The specific operation of BN is: first calculate $The\; mean\; and\; variance\; of\; B$ , after which$The\; mean\; and\; variance\; of\; the\; B$ set are transformed into 0 and 1, and finallythe BBEach element in$B$$\gamma$ plus$\beta$ , the output. $c$和$\beta$ is a trainable parameter that participates in the BP of the entire network;
the purpose of normalization is to regularize the data into a uniform interval, reduce the divergence of data, and reduce the learning difficulty of the network. The essence of BN is that after normalization, use$c$和$As\; a\; reduction\; parameter,\; \beta$ retains the distribution of the original data to a certain extent.
3. BBComposition of$B$

The dimensions of the tensor data passed in the neural network are usually recorded as [N, H, W, C], where N is batch_size, H and W are rows and columns, and C is the number of channels. Then the input set $B$ is the blue part in the figure below.

The calculation of the mean is to add up the numbers in each channel separately in a batch, and then divide by$N\times H\times w$ . For example: there are 10 pictures in this batch, each picture has three RBG channels, and the height and width of each picture are H and W, then the mean value is calculated by dividing the sum of the pixel values ​​​​of the R channels of the 10 pictures by 10$10\times H\times W$ , and then calculate the sum of all pixel values ​​of channel B divided by$10\times H\times W$ , finally calculate the sum of the pixel values ​​​​of the G channel divided by$10\times H\times w$ . The calculation of variance is similar.
Trainable parameter$c$和$The\; dimension\; of\; \beta$ is equal to the number of channels of the tensor. In the above example, each of the three channels of RBG needs a$\gamma$ and a$\beta$ , so$c$和$The\; dimension\; of\; \beta$ is equal to 3.
4. The mean and variance in BN during training and inference.
$During\; training,\; the\; mean\; and\; variance\; are\; respectively\; the\; mean\; and\; variance\; of\; the\; corresponding\; dimensions\; of\; the\; data\; in\; the\; batch;$
$During\; inference,\; the\; mean\; and\; variance\; are\; calculated\; based\; on\; the\; expectations\; of\; all\; batches\; and\; the\; method.\; The\; formula\; is\; as\; follows:$
$$E\left[x\right]\leftarrow E_B[m_B]$$ $$r\left[x\right]\_\leftarrow \frac{m}{m-1}{E}_B\left[p_B^2\right]$$
Reference:
https://zhuanlan.zhihu.com/p/93643523