The Backpropagation Principle and Derivation of Deep Neural Network DNN

DNN backpropagation derivation

---

1. Uniform symbols

As shown in the figure above, let's unify the symbols first:

$w$ heavy
$z$ input value
$b$ bias bais
$a$ activation value
$\sigma$ activation function

$w_{43}^2$Represents the weight b of the fourth neuron on the second hidden layer connected by the third neuron of the previous layer
$b_2^3$Represents the bias of the second neuron of the third hidden layer
$a_1^2$Indicates the activation value z of the first neuron on the second hidden layer
$z_1^2$Represents the input value of the first neuron on the second hidden layer

So there is

$$z_2^3=(a_1^2{w}_{21}^3+a_2^2{w}_{22}^3+a_3^2{w}_{23}^3+a_4^2{w}_{24}^3)+b_2^3$$
$$a_2^3=s\left(z_2^3\right)$$

---

2. Forward propagation

OK, we then replace the numbers with symbols:

$w_{jk}^l$Represents the weight bjl b_{j}^{l} of the jth neuron on the hidden layer of the first layer linked by the kth neuron of the previous layer
$b_j^l$Represents the bias ajl a_{j}^{l} of the jth neuron of the hidden layer of the l layer
$a_j^l$Indicates the activation value zjl z_{j}^{l} of the jth neuron on the hidden layer of layer l
$z_j^l$Indicates the input value of the jth neuron on the hidden layer of the lth layer

So we have:
$$z_j^l=\sum _k{w}_{jk}^l{a}_k^{l-1}+b_j^l$$
$$a_j^l=s\left(z_j^l\right)=s(\sum _k{w}_{jk}^l{a}_k^{l-1}+b_j^l)$$

Use vector display：
$$a^l=s(z^l)=s(w^{he}{to}^{l-1}+b^l)$$

---

3. Loss function

Suppose we use the mean square error MSE as the loss function
$$C=\frac{1}{2n}\sum _x\Vert y(x)-a^L(x){\Vert }^2$$

$a^L$ is the activation output of the last layer, L represents the last layer,$y\left(x\right)$ represents the Ground Truth when the input is x

We write it as a vector representation:
$$C=\frac{1}{2}\Vert y-a^L{\Vert }^2=\frac{1}{2}\sum _j(y_j-a_j^L{)}^2$$

Our goal is to optimize $C$ , making it the smallest.

---

4. Important note about BP

to $For\; C$ , the variable is not an input$x$ and$a$ , but$w$ . We're going to use$Cperforms\; gradient\; descent\; on\; the\; gradient\; ofw$ to change$w$ , in order to achieve one purpose, it is to obtain a set of$w$ such that for all input images or data$x$ makes$The\; averageC$ is the smallest. So actually what we are most concerned about in this problem isC vs. w C vs. wGradient of$C$$to$$w\; :$

$$\frac{\partial C}{\partial w_{jk}^l}$$
And $GradientofCtobias:$
$$\frac{\partial C}{\partial b_j^l}$$

BP backpropagation is relative to forward derivative calculation, so why use BP backpropagation to calculate these two gradients? The
answer is because the calculation is faster. Why the calculation is faster, let's first look at why the direct calculation of the gradient is slower.

Let's look at such a neural network with 2 hidden layers, we require $\frac{\partial C}{\partial w_{11}^2}$

$$\begin{gathered} \frac{\partial C}{\partial w_{11}^2}=\frac{\partial (loss(a_1^4)+loss(a_2^4))}{\partial w_{11}^2} \\ =\frac{\partial (loss(a_1^4\left)\right)}{\partial w_{11}^2}+\frac{\partial \left(loss\right(a_2^4\left)\right)}{\partial w_{11}^2} \\ =\frac{\partial C}{\partial a_1^4}\frac{\partial a_1^4}{\partial w_{11}^2}+\frac{\partial C}{\partial a_2^4}\frac{\partial a_2^4}{\partial w_{11}^2} \end{gathered}$$

Suppose we want to calculate $a_1^{4}The\; path\; toC$ , as shown by the red arrow,$w_{11}^2$All neurons affected are included in the computation

$$\frac{\partial C}{\partial a_1^4}easyto\; get$$

$$focuson\frac{\partial a_1^4}{\partial w_{11}^2}$$
由于$a_1^4=s\left(z_1^4\right)and{z}_1^4=a_1^3{w}_{11}^4+a_2^3{w}_{12}^4+a_3^3{w}_{13}^4+a_4^3{w}_{13}^4$所以,
$$\frac{\partial a_1^4}{\partial w_{11}^2}=\frac{\partial a_1^4}{\partial z_1^4}(\frac{\partial a_1^3{w}_{11}^4}{\partial w_{11}^2}+\frac{\partial a_2^3{w}_{12}^4}{\partial w_{11}^2}+\frac{\partial a_3^3{w}_{13}^4}{\partial w_{11}^2}+\frac{\partial a_4^3{w}_{13}^4}{\partial w_{11}^2})$$
OK, let's continue to push down, the red part in the above formula:
$$\begin{gathered} \frac{\partial a_1^3{w}_{11}^4}{\partial w_{11}^2} \\ =w_{11}^4\frac{\partial a_1^3}{\partial z_1^3}(\frac{\partial a_1^2{w}_{11}^3}{\partial w_{11}^2}+\frac{\partial a_2^2{w}_{12}^3}{\partial w_{11}^2}+\frac{\partial a_3^2{w}_{13}^4}{\partial w_{13}^3}) \end{gathered}$$
At this point we find that
$$\frac{\partial a_2^2{w}_{12}^3}{\partial w_{11}^2}and\frac{\partial a_3^2{w}_{13}^4}{\partial w_{13}^3}neitherw\_{}_{11}^{2}functionof\_所以都为0$$
于是,
$$\begin{gathered} \frac{\partial a_1^3{w}_{11}^4}{\partial w_{11}^2} \\ =w_{11}^4\frac{\partial a_1^3}{\partial z_1^3}(\frac{\partial a_1^2{w}_{11}^3}{\partial w_{11}^2}) \end{gathered}$$
So in the same way,

$$\begin{gathered} \frac{\partial a_1^4}{\partial w_{11}^2}=\frac{\partial a_1^4}{\partial z_1^4}(\frac{\partial a_1^3{w}_{11}^4}{\partial w_{11}^2}+\frac{\partial a_2^3{w}_{12}^4}{\partial w_{11}^2}+\frac{\partial a_3^3{w}_{13}^4}{\partial w_{11}^2}+\frac{\partial a_4^3{w}_{13}^4}{\partial w_{11}^2}) \\ =\frac{\partial a_1^4}{\partial z_1^4}(w_{11}^4\frac{\partial a_1^3}{\partial z_1^3}(\frac{\partial a_1^2{w}_{11}^3}{\partial w_{11}^2})+w_{12}^4\frac{\partial a_2^3}{\partial z_2^3}(\frac{\partial a_1^2{w}_{21}^3}{\partial w_{11}^2})+w_{13}^4\frac{\partial a_3^3}{\partial z_3^3}(\frac{\partial a_1^2{w}_{31}^3}{\partial w_{11}^2})+w_{14}^4\frac{\partial a_4^3}{\partial z_4^3}(\frac{\partial a_1^2{w}_{41}^3}{\partial w_{11}^2})) \\ =\frac{\partial a_1^4}{\partial z_1^4}\left(w_{11}^4\frac{\partial a_1^3}{\partial z_1^3}\right(w_{11}^3{x}_1)+w_{12}^4\frac{\partial a_2^3}{\partial z_2^3}(w_{21}^3{x}_1)+w_{13}^4\frac{\partial a_3^3}{\partial z_3^3}(w_{31}^3{x}_1)+w_{14}^4\frac{\partial a_4^3}{\partial z_4^3}(w_{41}^3{x}_1\left)\right) \\ =A \end{gathered}$$

$$\begin{gathered} \frac{\partial a_2^4}{\partial w_{11}^2}=\frac{\partial a_1^4}{\partial z_1^4}(w_{21}^4\frac{\partial a_1^3}{\partial z_1^3}(w_{11}^3{x}_1)+w_{22}^4\frac{\partial a_2^3}{\partial z_2^3}(w_{21}^3{x}_1)+w_{23}^4\frac{\partial a_3^3}{\partial z_3^3}(w_{31}^3{x}_1)+w_{24}^4\frac{\partial a_4^3}{\partial z_4^3}(w_{41}^3{x}_1)) \\ =B \end{gathered}$$

最后,
$$\begin{gathered} \frac{\partial C}{\partial w_{11}^2}=\frac{\partial C}{\partial a_1^4}\frac{\partial a_1^4}{\partial w_{11}^2}+\frac{\partial C}{\partial a_2^4}\frac{\partial a_2^4}{\partial w_{11}^2} \\ =\frac{\partial C}{\partial a_1^4}A \\ +\frac{\partial C}{\partial a_2^4}B \end{gathered}$$

At this point, the problem is basically clear.

We can see that every time a ∂ C ∂ wjkl \frac{\partial C}{\partial w_{jk}^l} is calculated during forward calculation$\frac{\partial C}{\partial w_{jk}^l}$must put $w_{jk}^l$Calculate the local partial derivatives on all paths to C. The number of links in a fully connected deep neural network is in the billions. It is very time-consuming and labor-intensive to calculate hundreds of millions of paths for every$gradient$$.$ .As can be seen from the above derivation,$A$和$Many\; calculations\; in\; B$ are repeated. Is there a way to calculate these repeated calculations in advance and then simply calculate the gradient through the relationship between the front and back layers? In this way, it is not necessary to search for all relevant path calculations.

Yes, that is BP. Of course, BP is just a mathematical technique for seeking partial derivatives, and it is a method rather than an end.

With the gradient, we only need to know the learning rate $\eta$ , then take all$w_{jk}^l-the\ast \frac{\partial C}{\partial w_{jk}^l}$A new set of weights is obtained. (The gradient is the fastest rising direction of the function, so here is a minus sign, the fastest falling direction.
) This set of weights will inevitably make $C$ decreases, and so on, until learning$\frac{\partial C}{\partial w_{jk}^l}$They are all approaching 0. In theory, they have reached a local bottom, and the learning of weights will slow down until they no longer change. In this way, the learning of the network is completed.

---

5.BP derivation

1）我们令
$$d_j^l=\frac{\partial C}{\partial z_j^l}$$
**Called BP error, Error, the essence of BP is it**

2) The error of the last layer:
$$d_j^L=\frac{\partial C}{\partial z_j^L}$$
由于：
$$a_j^L=s(z_j^L+b_j^L)$$

根据链式法则：
$$d_j^L=\frac{\partial C}{\partial a_j^L}\frac{\partial a_j^L}{\partial z_j^L}=\frac{\partial C}{\partial a_j^L}{p}^{\prime }(z_j^L)\text{(1)}$$

3) sinceδ
$$\begin{gathered} d_j^l=\frac{\partial C}{\partial z_j^l} \\ =\sum _k\frac{\partial C}{\partial z_k^{l+1}}\frac{\partial z_k^{l+1}}{\partial z_j^l} \end{gathered}$$
(The above formula needs to be savored carefully, especially why ${\sum }_k$Symbols, mainly to contain wwall paths affected by$w$
) and
$$d_j^{l+1}=\frac{\partial C}{\partial z_j^{l+1}}$$
则：
$$d_j^l=\sum _k\frac{\partial z_k^{l+1}}{\partial z_j^l}{d}_k^{l+1}\text{(2)}$$
So far, the relationship between the error of the previous layer and the next layer has surfaced

4) Next, let’s look at the C vs. w C vs. w we are most concerned aboutThe gradient of$C$$to$$w$$\frac{\partial C}{\partial w_{jk}^l}$And the gradient of C to bias $\frac{\partial C}{\partial b_j^l}$

由于
$$z_j^l=\sum _k{w}_{jk}^l{a}_k^{l-1}+b_j^l$$
则
$$\begin{gathered} \frac{\partial C}{\partial w_{jk}^l}=\frac{\partial C}{\partial z_j^l}\frac{\partial z_j^l}{\partial w_{jk}^l} \\ =d_j^l{a}_k^{l-1}\text{(3)} \end{gathered}$$

$$\begin{gathered} \frac{\partial C}{\partial b_j^l}=\frac{\partial C}{\partial z_j^l}\frac{\partial z_j^l}{\partial b_j^l} \\ =d_j^l\ast 1=d_j^l\text{(4)} \end{gathered}$$

So far we have $\left(1\right)\left(2\right)\left(3\right)\left(4\right)$ The four BP formulas are proved.

---

6. Summarize

when we have

$$d_j^L=\frac{\partial C}{\partial a_j^L}{p}^{\prime }(z_j^L)\text{(1)}$$

$$d_j^l=\sum _k\frac{\partial z_k^{l+1}}{\partial z_j^l}{d}_k^{l+1}\text{(2)}$$

$$\frac{\partial C}{\partial w_{jk}^l}=d_j^l{a}_k^{l-1}\text{(3)}$$

$$\frac{\partial C}{\partial b_j^l}=d_j^l\ast 1=d_j^l\text{(4)}$$

And because of the loss function:
$$C=\frac{1}{2}\Vert y-a^L{\Vert }^2=\frac{1}{2}\sum _j(y_j-a_j^L{)}^2$$
所以
$$\frac{\partial C}{\partial a_j^L}=(y_j-a_j^L)$$
We found that all the values ​​on the right side of the equal sign can be solved. The most amazing thing is that the gradient is only related to the error and the output of the previous layer. It is not necessary to calculate all the paths, and for a gradient descent, the δ of each$\delta$ only needs to be calculated once, and it can be used directly for calculation of other nodes after being saved, and the complexity is greatly reduced. So far, we have fully understood why BP is used and how BP is derived.