Challenge facing taught convolutional neural network comprises a gradient disappears, degradation, and an improved process comprising an improved convolution layer, a layer of the cell, activation function, loss of function, improvement of the network structure, the residual network, network-wide roll machine, multi- scale integration, batch normalization, etc.
Outline:
Challenges
gradient disappearing
degradation
general idea improvement
improvements convolution layer
realize the convolution matrix multiplication
improvement pooled layer of
improved activation function
improved loss function of
the motorway network
residuals network
analysis of network residuals
full volume network product
multiscale connection
batch normalize
this episode summary
Challenges:
Convolutional neural networks, especially the challenges faced by the depth of convolution neural network:
Gradient disappearing, fully connected neural network (also called on ANN artificial neural networks, multilayer perceptron model HLP) also mentioned, to use f when the propagation BP ' (X), if f ' (X) <0, will be more take smaller the final gradient tends to 0, the argument is not updated.
Degradation, refers to the network relatively shallow, as long as sufficient sample size to increase the number of layers of the network will increase the accuracy of the network, but instead add precision drops when the number of layers of the network if the network reaches a certain number of layers.
Overfitting, depth convolutional neural network is generally more layers, the width is relatively large, and the number of neurons per convolution is relatively large, so it is prone to overfitting.
Computing and storage efficiency, over the pursuit of precision network, the network of people to do more and more complex, very large width is also a great depth, large-scale network after, not only counted very slowly because of the number of operations will increase, and very take up storage space, which would limit its actual use.
Improved measures:
Convolution layer
cell layer
activation function
loss function
network structure
Gradient disappearing:
Deep Web training difficult, mainly due to the gradient disappearing
X. Glorot, Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. AISTATS, 2010
in the experiment were trained have 1-5 hidden layer of the neural network , activation function using sigmoid, tanh and other
experimental results show that, with the increase in the number of layers of the network, the role of counter-propagating smaller and more difficult to train the network
input value of the activation function easily fall into saturation interval, resulting in overfitting
Degradation:
Increase the number of layers of the network can improve the accuracy of the network, but then increased to a certain extent, with the increase in the level of training error and test neural network
test error increases, this problem is called degradation - similar dimension disaster
degradation and over-fitting different, over-fitting is poor accuracy on the test set, and degradation in the training set and test set accuracy has fallen
Improved general idea:
The goal is to improve the higher precision of the network, run faster.
Convolution layer
cell layer
activation function
loss function
network configuration
data normalization
Convolution layer improvements: