Backpropagation from zero learning error (gradient descent method)

1. Basic knowledge

The parameters in the convolutional neural network exist in the convolution kernel, and the convolution kernel can extract the features of the image, for example $\left[\begin{array}{ccc}1& 0& -1\\ 2& 0& -2\\ 1& 0& -1\end{array}\right]$The difference between the left and right sides of the left and right convolution kernel areas can be calculated. But for the recognition of the number "5" in the figure below, it is difficult for us to say whether the feature extracted by this convolution kernel is what we want, or whether this feature can allow the machine to recognize the number "5".

Deep learning can automatically learn these convolution kernels according to the loss function and data, so as to extract the best features that the network thinks.

1.1 Loss function

There are many kinds of loss functions, and the popular loss function is to measure the error between the supervised data and the output data, so as to close the distance between them.

y is the data output by the network, all of which add up to 1, indicating that the output has passed softmax.
t is the label data, and this label indicates that the correct class is the third class. Such a representation is called one-hot representation.

MSE

$MSE=\frac{1}{2}{\sum }_k(y_k-t_k{)}^2$
Calculated by this formula, the result is 0.0975

cross entropy error

$E=-{\sum }_k{t}_{k}log{y}_k$
Calculated by this formula, the result is 0.51

Although there is a large difference between the two values, in fact, the closer they are to the correct result, the smaller their values ​​are. In other words, as long as these two loss functions are made as small as possible, you will definitely get an answer that is the same as the label.

For example, in cross entropy, $y$越大，$The\; closer\; logy$ is to 0 (negative value), the closer to 0 overall.

1.2 Numerical differentiation

Numerical differentiation means to replace the denominator in the derivative formula with the smallest possible value (because it is practically impossible for the limit to approach 0), and a replacement will produce a certain error.

$f$ at$(x+h)$ and$(x-h)$ The difference between is called the central difference (such a difference) the error will be smaller

1.3 Partial derivatives

There is not only one variable in the partial derivative, but we need to distinguish when taking the derivative.
For example, $y=f(x_0,x_1)$ , which has two variables. as the picture shows.

It is also very simple to ask for partial derivatives. A three-dimensional surface can be seen in the figure. When a value is determined, for example,$x_0=0$ , the surface will become a curve, this curve is$f\left(x_1\right)$ curve. For the derivation of the curve, we will know it. At this time, we are seeking the partial derivation$\frac{\partial f}{\partial x_1}$。

1.4 gradient

The vector summed up by the partial derivatives of all variables is called the gradient.
For example, $y=f(x_0,x_1)$ , its gradient is ($\frac{\partial f}{\partial x_0},\frac{\partial f}{\partial x_1}$)

If you draw the gradient of each point of this function, it will be as shown in the figure.

It can be seen that they all point in one direction (this is the most idealized case), and this direction is the direction in which the value of the function decreases the most.
Why is this, suppose we have a function $f(x,y)$ , and an arbitrary direction$l$ .
The partial derivative (directional derivative) for this direction is shown in the figure.

$\alpha$和$\beta$ are respectively$l$ and$x$和$The\; included\; angle\; in\; the\; y$ direction,$cos$ is decreasing at [0,180], that is to say, when it is consistent with the gradient direction,$The\; value\; of\; cos$ is the largest, that is, the steepest. Similarly, the direction of the negative gradient is this steep slope (the direction in which the function value drops the fastest).
But in fact, even if you go down a steep slope at a certain point, you may not be able to reach the minimum value of the entire function (loss function) (the smallest and smallest difference will not be described here), but each point (each weight) It is a better strategy to move towards the direction where the function value will become smaller.

2. Gradient descent method

$$\begin{gathered} x_0=x_0-the\frac{\partial f}{\partial x_0} \\ {x}_1=x_1-the\frac{\partial f}{\partial x_1} \end{gathered}$$
$\eta$ represents the learning rate, that is, the magnitude of learning in each learning (epoch).
The above formula will be repeated with training, then$x_0$and $x_1$At the end substitute $It\; is\; very\; promising\; to\; reach\; the\; minimum\; value\; in\; f$ , then our purpose is completed.
code show as below:

import numpy as np

#定义一个函数
def function_1(x):
    f = x[0]**2 + x[1]**2
    return f

def n_d(f,x):
    h = 1e-4
    return (f(x+h) - f(x-h))/(2*h)

def numerical_gradient(f,x): #在数值偏导中，求x的偏导就给x增量
    h = 1e-4
    grad = np.zeros_like(x)
    #x很有可能有多个参数
    for i in range(len(x)):
        value = x[i]
        x[i] = value + h
        fxh1 = f(x)

        x[i] = value - h
        fxh2 = f(x)

        grad[i] = (fxh1 - fxh2)/(2*h)
        x[i] = value
    return grad


def gradient_descent(f,init,step):
    lr = 0.01
    x = init
    for i in range(step):
        grad = numerical_gradient(f,x)
        x -= lr*grad
    return x

init = np.array([-3.0,4.0])
res = gradient_descent(function_1,init,10000)
print(res)

2.1 Gradients of Neural Networks

Ok, so far the gradient of the binary function and its descent have been introduced. Let's analyze the gradient descent of the neural network.
For convenience, this formula is not typed out. As shown in the figure, there are only 2 unknowns -> 6 unknowns -> n, and the gradient is obtained in the same way.

3. Error Back Propagation Method

Although numerical differentiation is simple and easy to implement, its disadvantage is that it takes more time to calculate. We will learn a method that can efficiently calculate the gradient of weight parameters - error back propagation method.

3.1 The chain rule

z $$\begin{gathered} z=t^2 \\ t=x+y \end{gathered}$$

$z$ toxxThe derivative of$x$
$$\begin{gathered} \frac{\partial z}{\partial x}=\frac{\partial z}{\partial t}\frac{\partial t}{\partial x} \\ \frac{\partial z}{\partial x}=2t\_\ast 1=2(x+y) \end{gathered}$$
is expressed in the form of a computational graph, as shown in the figure The

initial signal of backpropagation is$\frac{\partial z}{\partial z}=1$ , as shown in the figure, when it reaches x, it will show ∂ z ∂ x \frac{\partial z}{\partial x}in a procedural form$\frac{\partial z}{\partial x}$the chain rule of

3.2 Backpropagation

When backpropagating, there is no need to care about the complex calculations that the input signal has undergone, and the output (the next reverse input) can be calculated by only focusing on the current node.
For example, the partial derivative of the addition node to the output unknown must be 1 (the premise is that only addition is involved, in fact, there must be only one kind of operation involved in the node), so as long as you go through the addition node, you only need Outputs the upstream value unchanged to the downstream.

The multiplication node considers $z=xy$ , multiplied by the flip value during forward propagation, as shown in the figure.

At present, we can give a calculation graph similar to the one below and fill in the correct numbers in the squares.

But in fact, the neural network does not know that addition and multiplication are so simple, and there are only a few that we can remember from memory. In addition to ordinary operations, there is also backpropagation of the activation layer.

3.2.1 Activation layer

Relu layer

$$y=\{\begin{array}{c}x(x>0)\\ 0(x\le 0)\end{array}$$
$$\frac{\partial y}{\partial x}=\{\begin{array}{c}1(x>0)\\ 0(x\le 0)\end{array}$$

Sigmoid layer

$$y=\frac{1}{1+exp(-x)}$$
Express this formula with a calculation graph, as shown in the figure (in fact, such a formula does not have only one fixed calculation graph representation).
After certain calculations, the answer in the figure below can be obtained.

Of course, you don't need to memorize any backpropagation formulas, just know his process. At present, we have learned about two methods for finding gradients, one is based on numerical solutions, and the other is based on analytical solutions. Obviously, the error back propagation method is based on analytical solutions.
Through the error back propagation method, the optimization of each step in the neural network can be known, so that the loss is continuously reduced.
For example, the above figure inputs x and outputs y. Then how to adjust x has been calculated $x=x-\frac{\partial L}{\partial y}y(1-y)$ .
After that, x is updated, and as x is updated, y is also updated. In this way, you can find the x that makes the sigmoid the lowest.

At this point we finally understand that the purpose of deep learning is to reduce or increase the loss, and error backpropagation can accomplish this task by adjusting x. ¹

---

1. Introduction to deep learning: theory and implementation based on python ↩︎