Forward calculation and backpropagation principle of fully connected neural network

Derivation of the weight update principle of the fully connected neural network (forward calculation + backpropagation)

In this question, we first introduced the basic mathematical principle of the weight update of the fully connected neural network - the gradient descent algorithm. Afterwards, this question will introduce the weight update principle of a single-layer perceptron (single-output fully connected neural network), so as to further explain the principle of weight update of a fully connected neural network.

1 Gradient descent algorithm

The gradient descent algorithm is an algorithm for finding the minimum value of a function. The gradient descent algorithm will be used to solve the minimum value of the objective function in the weight update later (the objective function will be explained later).

1.1 Gradient

The first thing to understand is: what is a gradient?
The gradient is a vector along which the function increases the fastest. The gradient contains the partial derivatives of the function with respect to the respective variables. For example for functions:

• $f(x,y)=x^2+y^2$

Given the exponential function:
$gradient=(\frac{x}{\sigma f(x,y)}，\frac{y}{\sigma f(x,y)})=(2x,2y)\_$

1.2 Gradient Descent Algorithm

Since the direction of the gradient is the direction in which the function increases the fastest, the opposite direction of the gradient must be the direction in which the function value decreases the fastest. Therefore, we can make the independent variable change in the opposite direction of the gradient, so that the function will gradually approach the minimum value (in fact, it is not the opposite direction of the gradient, as long as this direction is the direction in which the function value decreases).

Then, for example, for the function to be optimized $f(w,b)=w^2\ast x+b$
We can get the basic implementation steps of the gradient descent algorithm:

1. Initialize a $w_0$and $b_0$the value of
2. root $w_1=w_0-a_1\ast 2wx$，$b_1=b_0-a_2\ast 1$ (α is the learning rate, which can be 0.001) update$w$ andbbvalue of$b$
3. Keep updating until convergence

2. Weight update principle of single output perceptron

First define the naming rules of variables, as shown in the figure:

$w$ represents the weight, the subscript represents the numbers of the two connected nodes in their respective layers, and the superscript represents the number of connected upper layers.
$x$ indicates the input value, the subscript indicates the number of the input in the respective layer, and the superscript indicates the number of layers in the layer.
$O$ represents the value of the input value after the activation function, and$The\; x$ -like subscript indicates the number of the input in the respective layer, and the superscript indicates the number of layers in the layer.

The structure of a single-output perceptron is shown in the figure below:

The mathematical model of a single-layer perceptron is:

$x_1=x_0\ast w_0+x_1\ast w_1+x_2\ast w_2+...+x_n\ast w_n=\sum x_j\ast w_j$
$O_1=sigmoid(x_1)$
$E=\frac{1}{2}\ast (O_1-t_1{)}^2$

Among them, sigmoid is the activation function (a derivable activation function must be selected, otherwise the gradient cannot be obtained later); E is the error function, which is the object of optimization; t1 is the real value (label).
For a single-layer perceptron, our goal is: input a sample (including features from x0 to xn), and the model can output an accurate (or close to the real value) O1 value. So in a single-layer perceptron, we need weights $w_0,w_1,w_2,...,w_n$to update. Until the error function converges to the minimum value, the update is completed, $w_0,w_1,w_2,...,w_n$The value of is determined and used to predict new data.
So let's do the math and understand $w_0,w_1,w_2,...,w_n$The update process (the mathematical derivation part is temporarily skipped to see the results):
$w_j$against EEThe partial derivative formula of$E$
$$\frac{s{w}_j}{inE}=(O_0-t)O_0(1-O_0)x_j$$
Therefore, according to the gradient descent formula, $w_j$的手机最好的：
$$w_j=w_0-a\ast \frac{s{w}_j}{inE}=w_0-a\ast (O_0-t)O_0(1-O_0)x_j$$
Among them $w_0$Represents $w_{jk}$The initialization value.
That is, for each input sample $X$ (a sample includes features$w_0,w_1,w_2,...,w_n$），$w_0,w_1,w_2,...,w_n$Just update once. Therefore, if it is necessary to continuously update $w_0,w_1,w_2,...,w_n$, you need enough samples for training.

3. Weight update principle of multi-output perceptron

The schematic diagram of the multi-output perceptron is as follows:

Similar to the single-output perceptron, the mathematical model of the multi-output perceptron is:

$x_0^1=x_0^0\ast w_{00}^1+x_1^0\ast w_{01}^1+x_2^0\ast w_{02}^1+...+x_n^0\ast w_{0n}^1=\sum x_j^0\ast w_{jk}^1$
$O_0^1=sigmoid(x_0^1)$
$E=\sum _{k=1}^m\frac{1}{2}\ast (O_k^1-t_k{)}^2$

For multi-output perceptrons, the objective function is still $E$ , the objective functionEEThe independent variable of$E$$w_{00}^1,w_{01}^1,...,w_{0n}^1$, so for any weight $w_{jk}^0$, the gradient derivation process is similar.

For any $w_{jk}^0$, after inputting a sample and performing a forward calculation, $w_{jk}^0$A weight update will be performed:
$$w_{jk}^0=w_0-a\ast \frac{in}{s{w}_{jk}}=w_0-a\ast (O_k^1-t)O_k^1(1-O_k^1)x_j^0$$
Among them $w_0$means $w_{jk}^0$The initialization value.
Therefore, it is similar to the single-output perceptron: each time a multi-output perceptron inputs a sample, after a forward calculation and backpropagation, we can calculate all wjk 0 w_{jk}^ $w_{jk}^0$Perform a weight update. When enough samples are input and the weights are updated enough times, the predictive performance of the multi-output perceptron model will be strong enough (input a sample, and the multi-output perceptron can output a predicted value with a sufficiently small error from the true value).
Next, we can introduce the fully connected neural network model and its weight update principle by adding a hidden layer.

4. Weight update principle of fully connected neural network

The structure of the fully connected neural network is shown in the figure below:

between the input layer and the output layer, there are several hidden layers (here, for the convenience of analysis, only one hidden layer is drawn). Therefore, the input layer to the output layer are respectively I layer, J layer, and K layer.
For details, please refer to the forward calculation process of the multi-output perceptron. This section mainly derives the update process of the gradient ————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
: —————————————————————————————————————————————————————————————————————————————————————————————— Therefore, according to the above derivation: For the output layer node k∈K: σ E σ wjk K = ( O k K − tk ) O k K ( 1 − O k K ) O j J = δ k K ∗ O j J \frac{\sigma E}{\sigma w_{jk}^K}= (O_k ^K-t_k)O_k^K(1-O_k^K)O_j^J = \delta_{k}^K*O_j^J





$$\frac{in}{s{w}_{jk}^K}=(O_k^K-t_k)O_k^K(1-O_k^K)O_j^J=d_k^K\ast O_j^J$$
Wherein: $d_k^K=(O_k^K-t_k)O_k^K(1-O_k^K)$

对于隐含层节点j∈J：
$$\frac{in}{s{w}_{ij}^J}=O_j^J(1-O_j^J)O_i^I\sum _{k\in K}(O_k^K-t_k)O_k^K(1-O_k^K)=d_j^J\ast O_i^I$$
Among them: $d_j^J=O_j^J(1-O_j^J){\sum }_{k\in K}(O_k^K-t_k)O_k^K(1-O_k^K)$

For output layer node k∈K, $(O_k^K-t_k)O_k^K(1-O_k^K)$ is a variable that is only related to the K layer and the nodes after the K layer, so it is defined as$d_k^K$. Similarly, for the hidden layer node j∈J, $O_j^J(1-O_j^J){\sum }_{k\in K}(O_k^K-t_k)O_k^K(1-O_k^K)$Is a variable only related to J layer and nodes after J layer, so it is defined as $d_j^J$。

At the same time, we can find that when solving, only first solve $d_k^K$, in order to solve $d_j^J$. Similarly, when the number of layers increases, we can only solve the gradient update operator from the back to the front. Therefore, this process of updating weights is called backpropagation .

In actual programming, the following definitions can be made to facilitate the solution of the gradient (taking the kth layer as an example):
$$d^K=\left[\begin{array}{c}{d}_0^K\\ d_1^K\\ d_2^K\\ ...\\ d_m^K\end{array}\right]$$
$$O^J=\left[\begin{array}{ccccc}{O}_j^0& O_j^1& O_j^2& ...& O_j^n\end{array}\right]$$
Indicates,$\frac{in}{s{w}_{jk}^K}$According to 下式求解：
$$\frac{in}{s{w}^K}=d^{K.}{O}^J$$