Deep learning face questions 22: batch normalization in practice application

　　BN used in the convolution network

Assuming that the neural network has been trained with a good convolution operation of the BN, but when you use it to predict, often a time to enter a sample, then it passes through the network, meaning to calculate the mean and the variance is not big, often using the strategy is to calculate the mean and variance of the training phase of exponential moving average, and then use the mean and variance as BN when they operate in the forecast period.

Exponential Moving Average

Suppose the variable X _t over time t, exponential moving average that is defined according to the following rules

Assuming α = 0.7

When. 1 = T, X _{. 1} =. 5, the EMA ^{(. 1)} = X _{. 1} =. 5

When 2 = T, X ₂ = 10, the EMA ⁽²⁾ = [alpha] * EMA ^{(. 1)} + ([alpha]. 1-) * X ₂ =. 5 + 0.7 * (1-0.7) = 6.5 * 10

When. 3 = T, X _{. 3} = 15, the EMA ^{(. 3)} = [alpha] * EMA ⁽²⁾ + ([alpha]. 1-) * X _{. 3} = 6.5 + 0.7 * (1-0.7) = 9.05 * 15

When. 4 = T, X _{. 4} = 20 is, the EMA ^{(. 4)} = [alpha] * EMA ^{(. 3)} + (. 1-[alpha]) * X _{. 4} = 9.05 + 0.7 * (1-0.7) = 12.335 * 20 is

After four operations, the final moving average of 12.335

Corresponding to the code:

import numpy as np
import matplotlib.pyplot as plt
t = [1,2,3,4]
x = [5,10,15,20]
res = [x[0]]
for i in x[1:]:
    a = 0.7*res[-1]+0.3*i
    res.append(a)
plt.plot(t,x,"r")
plt.plot(t,res,"b")

View Code

For a bit more complicated exponential moving average of the observed image can be found, he will retain the original trend, and adapt to new trends:

import numpy as np
import random
import matplotlib.pyplot as plt
random.seed(20190725)
t = np.linspace(-5,5,100)
x = [-i**2+random.random()*15 for i in t]
res = [x[0]]
for i in x[1:]:
    a = 0.7*res[-1]+0.3*i
    res.append(a)
plt.plot(t,x,"r")
plt.plot(t,res,"b")