Return to Deep Learning Network model development history of classic catalog
Previous: depth articles - classic History of Deep Learning Network model (five) elaborate GoogleNet structure and characteristics (including v1, v2, v3)
Next: depth articles - classic History of Deep Learning Network model (seven) elaborate Inception-ResNet structure and features
In this section, ResNet elaborate structure and characteristics, the next section elaborate structure and characteristics Inception-ResNet
Paper Address: "Deep Residual Learning for Image Recognition"
II. Network Classic (Classic Network)
6. serious
ResNet launched by the Ho Kai Ming in 2015, swept all players on ISLVRC and COCO, won the championship. ResNet on the network structure has become a big innovation, rather than simply the accumulation of layers, but by the accumulation of residual module made of convolution neural network.
(1). With the deepening of the network, there has been training set accuracy rate of decline phenomenon, which is not due to over-fitting caused, but because of the network hierarchy deepened, when the back-propagation gradient due to diffusion. ResNet core idea is to introduce a so-called "quick connection identity" (Identity Shortcut Connection), skip one or more layers.
. (2) ResNet Neural Network Graphics wherein:
Some structures will increase ② or ④ layer behind a convolution kernel to change the number of channels, such that
the
dimensions are the same.
(3). ResNet chestnuts
(4). ResNet model Case
(5). ResNet characteristics
①. A depth learning network has a great influence on the effect of the final classification and identification, so the normal idea is to be able to network design deeper the better. But in fact not the case. Conventional network in the network stack deep, the effect is getting worse. One reason for that is: the deeper the network, the more obvious the phenomenon of the disappearance of the gradient, network training effect will not be very good. In the shallow network can not significantly improve the recognition performance of the network, the network consider how to deepen the gradient and will not disappear.
. ② ResNet initial inspiration from:
When deepening neural network, there will be a problem Degradation (degradation), that the accuracy rate will rise first, then reach saturation, and then continued to increase depth will result in decreased accuracy. This is not a problem of over-fitting, because not only increase the error rate on the test set, the training set itself the error rate increases. Suppose there is a relatively shallow network reaches saturation accuracy, then the back plus a few congruent mapping layer, at least the error will not increase, that is, the deeper the network should not bring rise on the training set error. The use of congruent mapping mentioned here directly to the output of the previous layer to the idea behind is the source of inspiration of ResNet.
③. Assuming that the input is a neural network certain desired output is a
, if the input passes directly to the output as the initial result, the goal is to learn this time
. ResNet is equivalent to changing the learning goals, learning is no longer a complete output
, but the difference between the input and output of
that residual. The original function of the need to learn
convert
. The Residual Connection Block Shortcut achieved by, for Element-wise in a superimposed by the Shortcut Block inputs and outputs of this, the simple addition does not give additional network parameters and computation, but at the same time can greatly increase Model speed training, improve the training effect, and when the deeper layers of the model, this simple structure can be a good solution to the problem of network degradation.
④. ResNet allows the original input information directly to the back layer.
⑤. ResNet residual network is built module. ResNet by a number that is a residual module stacked together to form a neural network depth.
⑥ biggest difference convolution neural networks and ordinary ResNet directly connected:
There are many ResNet bypass branch input layer is directly connected to the back, so that the back layer may be directly learn residuals.
⑦. Decline train a neural network with a gradient method, if no residuals will be found as deepening network, after training error is reduced to increase, while theoretically training error as small as possible. For the residual network, the number of layers increases, the training error is getting smaller and smaller, this approach allows the network to reach a deeper layer, the deep gradient help solve the problem of the disappearance of the network, so that training can only achieve a deeper networks, while ensuring a good performance.
⑧. Conventional convolutional layer or layers fully connected at the time the information is transmitted, there will be more or less information loss, loss is suppressed. ResNet in a way to solve this problem, a detour conducting output by directly entering information, to protect the integrity of information, the entire network is that part of the learning input, output difference of just simplify learning objectives and dimensions.
⑨. In ResNet paper, in addition to two proposed residual learning unit, also proposed a three-tier residual learning unit.
Return to Deep Learning Network model development history of classic catalog
Previous: depth articles - classic History of Deep Learning Network model (five) elaborate GoogleNet structure and characteristics (including v1, v2, v3)
Next: depth articles - classic History of Deep Learning Network model (seven) elaborate Inception-ResNet structure and features
