[Deep Learning] Basic Backpropagation Guidelines

This is actually a question that I have always wanted to record. I saw an easy-to-understand description of this in the book "Deep and Simple Neural Network and Deep Learning". Combined with the recently run routine, let's review it visually.

In the introduction of many books, backpropagation in deep learning is regarded as a black box. Users only need to know that backpropagation can be used to update weights to train the network. But the basic mechanism essence of this kind of training actually constitutes an interpretation of the neural network itself , and this interpretation is the key to understanding the applicability and scalability of the neural network, which is why this article decides to expand on it.

Network structure and parameter definitions

Before deriving the specific backpropagation mechanism, the definition of the network structure is first given.

Take a simple three-tier network structure as an example. The first layer is the input layer , here it is assumed that there are three input neurons; the second layer is the hidden layer (or middle layer), which means that the network is performing more complex feature extraction operations, the so-called deep network, that is, there are multiple hidden layers The neural network, the deepening of the network will improve its nonlinear expression ability; the third layer is the output layer , where the two activation values ​​output will compare the input cost function with the expected value and output the error result.

In the perceptron model in the previous section , we already know the connection weight $w$ , bias$b$ and the activation functionσ \sigmaThe activation value aacorresponding to$\sigma$The basic equations formed by$a\; .$For neural networks composed of multi-layer perceptrons, this equation for a single node remains the same. Using vector form, we will more concisely expressthe generation of activation values ​​​​of a certain layer:

$$a_l=s(w^{he}{to}^{l-1}+b^l)where\; the\; weighted\; inputz{}^l=w^{he}{to}^{l-1}+b^l$$

That is: the dot product of the connection weight between layers and the activation value of the upper layer plus the bias of this layer will get the weighted input $z^l$ , and the weighted input passes through the activation function to obtain the activation value of the layer.

Supplement a matrix operation symbol Adamar product or Schur product $\odot$ , which subsequently simplifies the writing of the equation.

assuming $s$ and$t$ is two vectors with the same dimension, then$s\odot t$ represents the element-wise product, ie$(s\odot t{)}_j=s_j{t}_j$

Examples are as follows:

$$\left[\begin{array}{c}1\\ 2\end{array}\right]\odot \left[\begin{array}{c}3\\ 4\end{array}\right]=\left[\begin{array}{c}1\ast 3\\ 2\ast 4\end{array}\right]=\left[\begin{array}{c}3\\ 8\end{array}\right]$$

derivation of the equation

In fact, backpropagation considers how to change the weight and bias to control the cost function , and its ultimate meaning is to calculate the partial derivative $\partial C/\partial w_{jk}^l$ 和 $\partial C/\partial b_j^l$. An enlightening realization - $\partial C/\partial z_j^l$is a measure of neuron error (here choose $z_j^l$The expression error is a simplification for the subsequent equation form), that is, the definition of the $l$ layerjjError δ jl \delta _{j}^{l}$on\; j$ neurons$d_j^l$Definition:
$$d_j^l\equiv \frac{\partial C}{\partial z_j^l}$$

By convention, use $d^l$ means the same as thellThe error vector associated with layer$l\; .$Each layer of δ l \delta^{l}can be calculated using backpropagation$d^l$ , and then compare these errors with the actual required amount$\partial C/\partial w_{jk}^l$ 和 $\partial C/\partial b_j^l$connect.

Before starting the proof process, first give four formulas about the backpropagation equation , as follows:

Summary: Backpropagation Equation

$$d^L={\nabla }_{a}C\odot p^{\prime }(z^L)output\; layer\; error(1)$$

$$d^l=((w^{l+1}{)}^T{\delta }^{l+1})\odot p^{\prime }(z^l)front\; pass\; error(2)$$

$$\frac{\partial C}{\partial b_j^l}=d_j^{l}Bias\; rate\; of\; change(3)$$

$$\frac{\partial C}{\partial w_{jk}^l}=a_k^{l-1}{d}_j^{l}rate\; of\; change\; of\; weight(4)$$

Let's start to prove one by one, please remember the core multivariate calculus chain rule .

(1) Output layer error

For the output layer error, it is defined as:
$$d_j^L=\frac{\partial C}{\partial z_j^L}$$
Among them, L is the number of layers of the network, which means that this is the error on the jth neuron of the output layer . In fact, we can rewrite the partial derivative above in terms of the partial derivative of the output activation value, that is,
$$d_j^L=\sum _k\frac{\partial C}{\partial a_k^L}\frac{\partial a_k^L}{\partial z_j^L}$$
where the summation is performed over all neurons in the output layer. Of course, the $The\; output\; activation\; value\; of\; k$ neurons$a_k^L$only depends on $k=$The input weight zj of the jth neuron at time$j$$z_j^L$, so when $k\ne j$时$\partial a_k^L/\partial z_j^L$Does not exist (i.e. no impact is 0). Therefore, the equation can be reduced to
$$d_j^L=\frac{\partial C}{\partial a_j^L}\frac{\partial a_j^L}{\partial z_j^L}$$
This shows that the error of the neuron is only related to the partial derivative of the loss function to the activation value of the neuron and the partial derivative of the activation value to the weighted input on the neuron (that is, the derivative value of the activation function at this point). Based on $a_j^L=s\left(z_j^L\right)$ , the second item on the right can be written as
$$p_j^L=\frac{\partial C}{\partial a_j^L}{p}^{\prime }(z_j^L)$$
Extend the above formula to each neuron in the output layer, and use the Hadamard product to express, you can get the form of formula (1), that is,
$$d^L={\nabla }_{a}C\odot p^{\prime }(z^L)$$

(2) Front layer transfer error

This should be the formula that best explains the meaning of the word "reverse" in the backpropagation algorithm, that is, the error is passed from the back to the front. Specifically, the consideration is to use the next layer of error $d^{l+1}$ to represent the error$d^l$ . Similarly, we use$d_k^{l+1}=\partial C/\partial z_k^{l+1}$重写$d_j^l=\partial C/\partial z_j^l$Let
$$\begin{array}{rl}{d}_j^l& =\frac{\partial C}{\partial z_j^l}\\ & =\sum _k\frac{\partial C}{\partial z_k^{l+1}}\frac{\partial z_k^{l+1}}{\partial z_j^l}\\ & =\sum _k\frac{\partial z_k^{l+1}}{\partial z_j^l}{d}_k^{l+1}\end{array}$$
The last line swaps the two terms on the right and substitutes $d_k^{l+1}$The definition of , correspondingly realizes the substitution of the lower layer error. To evaluate the last row, expand
$$z_k^{l+1}=\sum _j{w}_{kj}^{l+1}{a}_j^l+b_k^{l+1}=\sum _j{w}_{kj}^{l+1}p\left(z_j^l\right)+b_k^{l+1}$$
Differentiate it to get
$$\frac{\partial z_k^{l+1}}{\partial z_j^l}=w_{kj}^{l+1}{p}^{\prime }\left(z_j^l\right)$$
back to the original formula, there will be
$$d_j^l=\sum _k{w}_{kj}^{l+1}{d}_k^{l+1}{p}^{\prime }\left(z_j^l\right)$$
This is also the component form of (2). After introducing the Hadamard product operator, the vector form of transferring the error from the back layer to the front layer can be written, that is, δ l = ( ( wl + 1 ) T δ l +
$$d^l=((w^{l+1}{)}^T{\delta }^{l+1})\odot p^{\prime }(z^l)$$

(3) Rate of change of bias

Similar to the proofs of the previous two steps, we can imagine that the proof at this time is still unfolded by applying the chain rule. Considering
$$d_j^l=\frac{\partial C}{\partial z_j^l}=\frac{\partial C}{\partial b_j^l}\frac{\partial b_j^l}{\partial z_j^l}$$
Among them, zjl z_{j}^{l} can be$z_j^l$展开
$$z_j^l=\sum _k{w}_{jk}^l{a}_k^{l-1}+b_j^l$$
Both sides are facing $b_j^l$Find the partial derivative, because $w_{jk}^l$with $b_j^l$irrelevant, because each is a variable that can change freely. So
$$\frac{\partial z_j^l}{\partial b_j^l}=1$$
Then, there is the proof of (3).
$$\frac{\partial C}{\partial b_j^l}=d_j^l$$

(4) Rate of change of weight

Similar to the proof in (3), we have
$$d_j^l=\frac{\partial C}{\partial z_j^l}=\frac{\partial C}{\partial w_{jk}^l}\frac{\partial w_{jk}^l}{\partial z_j^l}$$
Same as above, General $z_k^l$展开
$$z_j^l=\sum _k{w}_{jk}^l{a}_k^{l-1}+b_j^l$$
Both sides at the same time to $w_{jk}^l$Find the partial derivative, $w_{jk}^l$Not only with $b_j^l$has nothing to do with other weights, but only with itself, so we have:
$$\frac{\partial z_j^l}{\partial w_{jk}^l}=a_k^{l-1}$$
After transformation, there is a proof of (4)
$$\frac{\partial C}{\partial w_{jk}^l}=a_k^{l-1}{d}_j^l$$

summary

Through the above proof process, we can easily see how we pass back the error layer by layer through the backpropagation algorithm, and update the mathematical principles of weights and biases.

Now we look back at equation (1), and in its component form, look at the term $p^{\prime }(z_j^L)$ . Looking back at the graph of the sigmoid function, when$s\left(z_j^L\right)$ is approximately 0 or 1, the sigmoid function becomes very smooth, then$p^{\prime }(z_j^l)\approx 0$ . Therefore, if the output neuron is at a small activation value (about 0) or a large activation value (about 1), the weight learning of the final layer will be very slow. At this time, it can be said that the output neuron is saturated, and the weight learning isalso will terminate (or learn very slowly), and the output neuron will be biased similarly. Because we look back at equations (3) and (4), we can find that the update of weights and biases actually depends on the error$s\left(z_j^l\right)$ , and for the output layer, the error depends on the derivative of the activation function$p^{\prime }\left(z_j^L\right)$ , so in a saturated state, learning becomes difficult.

Let's look at (2) finally, notice the term $p^{\prime }\left(z_j^l\right)$ , which means that the error will be multiplied by the derivative value of an activation function when it is passed back to the front layer. For the sigmoid function, we can know that the derivative value is between 0 and 1. If the number of layers of the network is large, when it propagates from the output layer to the shallow layer, we can know that the error will be reduced many times, and the error will eventually Will drop to a negligible level (close to 0), so shallow learning will become very difficult, which is thegradient disappearance.

Also, consistent with the analysis in the output layer neuron saturation scenario, the saturation of interneurons will also lead to $d_j^l$becomes small because $p^{\prime }(z_j^l)$ is very small. That is to say:learn slowly for any weight input to a saturated neuron.

The backpropagation equation is valid for any activation function. The discussion of the cost function is ignored here, because that is another piece of content, but it can be proved that the above inference itself has nothing to do with any specific cost function . At the same time, we may have new ideas, and the sigmoid function may not be suitable as an ideal activation function in some applications. In fact, designing an activation function with specific learning properties is actually an important part of the neural network .

try to observe

Here is a simple example to illustrate how to specifically observe the iterative update of weights in a neural network. This is an example of using a neural network for nonlinear fitting.

First we generate a noisy sinusoidal signal (non-linear signal), as follows:

x = torch.unsqueeze(torch.linspace(-np.pi, np.pi, 100), dim = 1) # 构建等差数列
y = torch.sin(x) + 0.5*torch.rand(x.size())  # 条件随机数

Then we define a network structure:

class Net(nn.Module):  # 定义类，存储网络结构
    def __init__(self):
        super(Net, self).__init__()
        self.predict = nn.Sequential(
            nn.Linear(1,10),  # 全连接层，1个输入，10个输出
            nn.ReLU(),  #ReLU激活函数
            nn.Linear(10,1)  # 全连接层，10个输入，1个输出
        )

The activation function used here is the ReLU function. Compared with the Sigmoid function, ReLU does not have the problem of gradient disappearance in the positive interval, and the function derivation is relatively simple . It is a commonly used activation function in CNN networks.

During training, we can obtain the weights corresponding to the fully connected layer by calling modules and traversing the layers. The corresponding code is as follows:

        # 输出权重变化，net.modules返回的是generator类型
        for layer in net.modules():
            if isinstance(layer, nn.Linear):
                print(layer.weight)

The code of the whole process is as follows:

# 这是一个用神经网络进行函数拟合的函数
import  numpy as np
import torch
import torch.nn as nn
import matplotlib.pyplot as plt

x = torch.unsqueeze(torch.linspace(-np.pi, np.pi, 100), dim = 1) # 构建等差数列
y = torch.sin(x) + 0.5*torch.rand(x.size())  # 条件随机数

class Net(nn.Module):  # 定义类，存储网络结构
    def __init__(self):
        super(Net, self).__init__()
        self.predict = nn.Sequential(
            nn.Linear(1,10),  # 全连接层，1个输入，10个输出
            nn.ReLU(),  #ReLU激活函数
            nn.Linear(10,1)  # 全连接层，10个输入，1个输出
        )

    def forward(self,x):  # 定义前向传播过程
        prediction = self.predict(x)  # 将x传入网络
        return prediction


net = Net()
optimizer = torch.optim.SGD(net.parameters(), lr = 0.05)  # 设置优化器
loss_func = nn.MSELoss()  # 设置损失函数
plt.ion()   # 打开交互模式

for epoch in range(1000):
    out = net(x)   # 实际输出
    loss = loss_func(out, y)  # 实际输出和期望输出传入损失函数
    optimizer.zero_grad()  # 清除梯度
    loss.backward()  # 误差反向传播

    optimizer.step()  # 优化器开始优化
    if epoch % 25 == 0:  # 每25epoch显示
        plt.cla()   # 清除上一次绘图
        plt.scatter(x,y)  # 绘制散点图

        # 输出权重变化，net.modules返回的是generator类型
        for layer in net.modules():
            if isinstance(layer, nn.Linear):
                print(layer.weight)

        plt.plot(x, out.data.numpy(), 'r', lw = 5 )  # 绘制曲线图
        plt.text(0, 0 , f'loss={
      
      loss}',fontdict={
    
    'size':20,'color':'red'})  #  添加文字来显示loss值

        plt.pause(0.1) # 显示时间0.1s

    plt.show()

plt.ioff()  # 关闭交互模式
plt.show()  # 定格显示最后结果

You can see the fitting effect displayed in real time on the graphical interface. As the loss value decreases, the network will gradually fit the shape of the sinusoidal curve.

We can also see the output network weights in the console. Here, the weight obtained during the last training is taken as an example. You can see a weight matrix of 10×1 (1→10) and a weight matrix of 1×10 (10→1), which are different from the previous definition. The network structure of the two fully connected layers of is exactly corresponding.

---

Reference:
"Python Neural Network and Deep Learning" Author: [Australia] Michael Nielsen Translator: Zhu Xiaohu Chapter 2
"Python Neural Network Introduction and Practical Combat" Author: Wang Kai Chapter 7