Two-layer and N-layer fully connected neural network model principle

foreword

  Deep learning is to learn the internal laws and representation levels of sample data. The information obtained during the learning process is of great help to the interpretation of data such as text, images and sounds. Fully connected neural network (MLP) is one of the basic network types, which fully embodies the characteristics of deep learning methods compared with traditional machine learning algorithms, namely big data drive, formula derivation, self-iterative update, black box training, etc. This article will analyze and explain MLP from two layers and above.

1. Two-layer MLP

  The difference between a two (shallow) layer neural network and a single layer network is that the hidden layer has multiple neural nodes, which allows it to deal with the complex problem of " multiple input and multiple output ".

1.1 Forward Propagation

$$\mathrm{y}(x,W)=Wx+b$$
  where$x$ represents the input image, its dimension is$d$; $y$ is a score vector whose dimension is equal to the number of categories$c$；$W=[w_1\cdot \cdot \cdot w_c{]}^T$为权值矩阵，$w_i=[w_i\cdot \cdot \cdot w_{id}{]}^T$ is theiiThe weight vector of$i$$b=[b_i\cdot \cdot \cdot b_c{]}^T$ is the bias vector,$b_i$for $The\; bias\; of\; i$ categories, then the two-layer MLP is
$$y=W_2\sigma (0,W_{1}x+b_1)+b_2$$, where $\sigma$ is the activation function

1.2 Backpropagation

  What backpropagation does is let each neuron have a $The\; values\; of\; Wandb$ , that is, the gradient. In this way, when we pass in a new data, we can accurately predict it, and of course it is also a feedback on the data propagated from each layer. When the data is fed back, the loss function is the evaluation method. The following will take themean square error loss functionas an example to descend its gradient.

  损失函数：$$L(\hat{y},y)=\frac{1}{2}({\hat{y}}_i-y_i{)}^2$$

  梢度 名生: $$\begin{array}{l}{w}_1=w_0-the\frac{dL(w)}{dw}\\ \\ b_1=b_0-the\frac{dL(b)}{db}\end{array}$$

  Among them $w_0$and $b_0$is our current actual value, $-\eta$ isthe step size(certain value), when$L$ takes extreme value$w$ time,$w_1$Is the value obtained by gradient descent.

  When the gradient of the loss function is reduced, the chain rule needs to be solved

$$\frac{dL(a,y)}{dw}=\frac{dL(a,y)}{da}\cdot \frac{d}{dz}\cdot \frac{dz}{dw}$$
  Deduction :
  Gradient descent into the loss function

  chain rule

  final result

2. N-tier MLP

  N-layer fully connected neural network - a network with N number of layers other than the input layer.
  In a neural network, as the number of layers in the network increases, each layer abstracts the previous layer deeper. Each layer of neurons learns a more abstract representation of the values ​​of neurons in the previous layer.Three-layer neural network is also called two hidden layer neural network，Three-layer MLP is: $$y=W_3\sigma (0,W_2\sigma (0,W_{1}x+b_1)+b_2)$$ , where$\sigma$ is the activation function.

2.1 Network parameters

  Parameter: Refers to the algorithm running iterations and correcting the final stable value.

  Hyper-parameters: network structure - the number of neurons in the hidden layer, the number of network layers, nonlinear unit selection, etc.
     Optimization related - learning rate, dropout ratio, regularization term strength, etc.

2.2 Hyperparameter optimization

  Grid search method :

    ① Take several values ​​for each hyperparameter, and combine these hyperparameter values ​​to form multiple sets of hyperparameters; ②Evaluate the
    model performance of each group of hyperparameters on the validation set;
    ③Select the set of values ​​​​used by the model with the best performance as the final hyperparameter value.

  Random search method :

    ① Randomly select points in the parameter space, and each point corresponds to a set of hyperparameters;
    ② Evaluate the model performance of each set of hyperparameters on the verification set;
    ③ Select the set of values ​​​​used by the model with the best performance as the final hyperparameters value.

  Hyperparameter search strategy :

    ①Coarse search: Use the random method to sample hyperparameters in a large range, train for one cycle, and narrow the range of hyperparameters based on the accuracy of the verification set.
    ②Fine search: Use the random method to sample hyperparameters within the aforementioned narrowed range, run the model for five to ten cycles, and select the set of hyperparameters with the highest accuracy on the verification set.

3. MLP optimization

  Non-linear factors: Around the activation function, to increase the calculation rate, the activation function must be de-integrated, de-differentiated, and easy to obtain partial derivatives, so as to solve the problems of gradient disappearance and gradient explosion;

  Iterative update: update weights and biases around backpropagation, loss function selection, optimizer selection, learning rate decay strategy, etc.;

  Backbone network: How many layers should be set up in the network, and how many nodes should each layer have.

  The above is the principle of the MLP model of two layers and N layers (taking three layers as an example). For MLP optimization, you can refer to the optimization and improvement of the fully connected neural network in this column.