Neural Networks: Fundamentals of Deep Learning

1. Concept and simple derivation of backpropagation algorithm (BP)

The Backpropagation (BP) algorithm is a common algorithm used in conjunction with optimization methods (such as gradient descent) to train artificial neural networks . The BP algorithm calculates the gradient of the loss function for all weights in the network and feeds the gradient back to the optimization method to update the weights to minimize the loss function. The algorithm first calculates (and caches) the output value of each node through forward propagation, and then calculates the partial derivative of the loss function value with respect to each parameter through back propagation through the graph .

Next, we take a fully connected layer, a neural network using sigmoid activation function, and Softmax+MSE as the loss function as an example to deduce the logic of the BP algorithm. Due to space limitations, only a simple derivation will be carried out here. Rocky will write a dedicated article on the complete derivation process of the PB algorithm in the future, so stay tuned.

First, let's look at the expression of the sigmoid activation function and its derivatives:

$\sigma(x) = \frac{1}{1+e^{-x}}$
$sigmoid导数：\frac{d}{dx}\sigma(x) = \sigma(x) - \sigma(x)^2 = \sigma(1- \sigma)$

It can be seen that the derivative of the sigmoid activation function can ultimately be expressed as a simple operation on the output value.

Let’s look at the expression of the MSE loss function and its derivatives:

$\frac{1}{2}\sum^{K}_{k=1 }(y_k - o_k)^2$

Among them $y_k$ Represents the ground truth (gt) value, $o_k$ Represents the network output value.

$\frac{\partial L}{\partial o_i} = (o_i - y_i)$

Since the partial derivative is simple and only if $k = It will only work if i$ , so it is simplified.

Next, let’s look at the gradient output by the fully connected layer:

$\frac{1}{2}\sum^{K}_{i= 1}(o_i^1 - t_i)^2$

$\frac{\partial L}{\partial w_{jk}} = (o_k - t_k)o_k(1-o_k)x_j$

We use $\delta_k = (o_k - t_k)o_k(1-o_k)$ , it can be simplified again:

$\frac{dL}{dw_{jk}} = \delta_kx_j$

Finally, let’s look at the partial derivatives of each layer in the PB algorithm:

output layer:
$\frac{\partial L}{\partial w_{jk}} = \delta_k^K o_j$
$\delta_k^K = (o_k - t_k)o_k(1-o_k)$

Penultimate layer:
$\frac{\partial L}{\partial w_{ij}} = \delta_j^J o_i$
$\delta_j^J = o_j(1 - o_j) \sum_{k}\delta_k^Kw_{jk}$

Third layer of inverted numbers:
$\frac{\partial L}{\partial w_{ni}} = \delta_i^I o_n$
$\delta_i^I = o_i(1 - o_i) \sum_{j}\delta_j^Jw_{ij}$

By deriving it backwards like this, and then iteratively optimizing the network parameters through the gradient descent algorithm, you can complete the PB algorithm logic.

2. Related concepts of moving average

The moving average (exponential moving average), or exponentially weighted moving average (exponentially weighted moving average), can be used to estimate the local mean of a variable, so that the update of the variable is related to the historical value over a period of time .

variablevv $vattt$ _Time $t$ $v_{t}$ ， $\theta_{t}$ for the variable $vattt$ _ $The value after training at time t$ , when the moving average model is not used, $v_{t} = \theta_{t}$ , after using the moving average model, $v_{t}$ The update formula is as follows:

In the above formula, $\beta\epsilon[0,1)$ 。 $\beta = 0$ is equivalent to not using a moving average.

variable $at time t$ The sliding average of $v$ $1/(1-\beta)$ times $\theta$ values. And use bias correction to change $v_{t}$ Divide by $\beta^{t})$ corrects the estimate of the mean.

After adding Bias correction, $v_{t}$ 和 $v_{biased_{t}}$ The update formula is as follows:

when $The larger t$ is, $\beta^{t}$ is to 1, the results obtained by formulas (1) and (2) ( $v_{t}$ 和 $v_{biased_{1}}$ ) will get closer and closer.

This $\beta$ is, the closer the value obtained by the sliding average is to $\theta$ Related to historical values of $θ .$ If $\beta = 0.9$ , which is roughly equal to the past 10 $\theta$ values; if $\beta = 0.99$ , which is roughly equal to the past 100 $\theta$ values.

The figure below represents the results of calculating weights in different ways:

As shown in the figure above, the sliding average can be regarded as the average value of the variable in the past period. Compared with the direct assignment of the variable, the value obtained by the sliding average is smoother and smoother on the image, with less jitter, and will not be affected by Certain abnormal values cause the sliding average to fluctuate greatly .

The advantage of sliding average: it takes up less memory and does not need to save the past 10 or 100 histories $\theta$ can be estimated as its mean value. Although the sliding average is not as accurate as calculating the average by saving all historical values, the latter takes up more memory and is more computationally expensive.

Why is the sliding average used during testing?

Moving average can make the model more robust on test data .

When using the stochastic gradient descent algorithm to train a neural network, using a sliding average can improve the performance of the final model on test data to a certain extent in many applications.

The weights of the neural network during training $w e i g h t s$ after the moving average during the test $w e i g h t s$ is used as the weight during testing, which has better effect on test data. after the moving average $The update of w e i g h t s$ is smoother. For stochastic gradient descent, the smoother update means that it will not deviate very far from the optimal point. For example, assuming decay=0.999, a more intuitive understanding is that during the last 1,000 times of training, the model has already been trained and is in the jitter stage, and the sliding average is equivalent to averaging the last 1,000 jitters, so that The weights will be more robust.