Basic network review (Basemodel)

Here is what I in July 2018 when looking for a job, review (Basemodel) based on individual learned summed up the underlying network.
Among them, only pick I think the more important series of basic network, a brief overview in chronological order.

If elaborated wrong place, kindly pointed out.

LeNet

time

1998

significance

1. CNN's launch marks the real.

advantage

1. It achieved a 99% accuracy rate higher than on character recognition.

Shortcoming

1. There was no GPU, training LeNet very time-consuming;
2. Non-OCR tasks, even worse than SVM.

application

1. OCR (such as the US postal system).

AlexNet

time

2012

significance

1. It demonstrates the effectiveness of CNN in a complex model, CV areas of great development "blowout."
2. ImageNet2012 1st.

Innovation

• Data levels:
  1. ImageNet using mass data (to prevent over-fitting);
  2. Use the data enhancement (to prevent over-fitting).
• The network level:
  1. Was added to prevent overfitting dropout;
  2. Relu activation function, avoid the upside of the gradient disappears;
  3. Partial response normalized (LRN), use the data to do near normalized (bn appeared after 2015 phase-out).
• Hardware level:
  1. Using the GPU, rapid convergence (double pipe-line represents the paper, i.e. parallel dual GPU).

advantage

Breaking the ceiling layers, depth of eight.

application

1. R-CNN；
2. SPPNet。

ZFNet

time

2013

significance

1. Using deconvolution of CNN visual features, CNN's insight into the hierarchy.

application

1. Faster R-CNN。

the

significance

1. Multilayer perceptron (fully connected multilayer composition layer and the nonlinear function) replaces the previous CNN simple linear convolution layer. Because of the nonlinear depth study of all abstraction and representation capability of the source, and therefore the equivalent of "Gasser MLP between layers to increase non-linear."

application

1. Of reference by Inceptionv1.

GoogleNet

time

2014

significance

1. Proposed Inceptionv1, opened the Inception series;
2. Refresh the depth of the network set a new record depth of the new network;
3. The first proposed method of dimension reduction is conv1 × 1. The first proposed method of dimension reduction is conv1 × 1.

Innovation

1. NiN similar structure, i.e. the original node is a sub-network.

advantage

1. Parameter less.

Shortcoming

1. complex structure.

variant

1. Inceptionv2/3/4、Xception

application

1. YOLOv1。

VGGNet

time

2014

significance

1. ImageNet2014 positioning tasks first, second classification task.

Innovation

1. The widespread use of small network convolution kernel to replace the large convolution kernel.

advantage

1. Network architecture super-simple, highly modular.
2. Convolution kernel can be achieved with the same large scale of the receptive field after a small convolution kernel stack, and a smaller amount of parameters;
3. Deepened network layers -> enhances network capacity.

Shortcoming

1. Model too (most of the parameters in the last three fc layers), inefficient.
2. Superposition convolution kernel means that the number of layers increases, leading to an increase in total calculated amount of the entire network;
3. Increase the number of layers leads to greater risk gradient.

structure

Convolution five scale, the first two layers each containing 2 convolution, the convolution of three layers, each containing 3-4; FC followed by three, the last ending Softmax.

application

1. Fast R-CNN；
2. Faster R-CNN；
3. SSD。

serious

time

2015

significance

1. The current mainstream basemodel;
2. ImageNet2015 champion;
3. CVPR2016 Best Paper。

Innovation

1. Highway Networks first design of mapping between the different layers, and ResNet Highway Networks simplified design, mapping performed only between adjacent module. Once again, the gradient eased the problem, and once again broke the network layers of the ceiling. Meanwhile, the design also makes early training can be extremely fast convergence.

analysis

Why Effective:

1. Model to learn, there has always been a part of the structure of identity mapping. The original "serial" network is difficult to learn this mapping. ResNet representing an increase of constraints to help model easily learned the identity mapping.
2. After adding identity mapping, learning objectives into a residual, it is significantly easier to learn than the original object.
3. Standing identity mapping perspective, then deep ResNet fact, not several layers.

Think

1. ResNet more than 50 layers have used conv1 × 1, it is to think dimensionality reduction.

variant

1. ResNeXt；
2. DenseNet;
3. DPN.

application

1. R-FCN;
2. FPN(+Faster R-CNN)；
3. Mask R-CNN；
4. RetinaNet。

DenseNet

time

2016

significance

1. CVPR2017 Best Paper。

Innovation

1. The original ResNet of "serial type one identity mapping" turned into "identity mapping one to many", the design is relatively simple.

Think

1. Because it is only for identity mapping ResNet be carried forward, this idea a lot of people can think of, all the text is not amazing.

application

1. Play brush list;
2. Paper brush AP.

Xception

time

2016

significance

1. The most mainstream of several lightweight one basemodel.

Innovation

1. Group replaces a serial parallel type previously used group. (The group responsible for the serial module is called separable convolution)

advantage

1. Parameter was minimal, ultra-small model.

Shortcoming

1. Feature extraction capability is poor.

variant

1. Xception145；
2. Xception39。

ResNeXt

time

2016

Innovation

The original ResNet simple "residual architecture" replaced by "Inception version of the residual structure."

advantage

On ImageNet can converge faster than the ResNet classification results slightly better point.

Shortcoming

complex.

application

1. Play brush list;
2. Paper brush AP.

DPN

time

2017

Innovation

1. Double pipe-line. One side is ResNet, one side is DenseNet, known as the pipe-line allows the two complement each other.

application

1. Play brush list;
2. Paper brush AP.

---

to sum up

Before ResNet out, basemodel gone from a dominant AlexNet, to GoogleNet and VGGNet split the world period. After ResNet out due to its simplicity and powerful, become the absolute benchmark of basemodel.

Today, practitioners follow the general process is:

1. First with ResNet-50 to verify the effectiveness of the algorithm;
2. When this algorithm is effective in terms ResNet-50, if to pursue speed of the algorithm (e.g., ground to a mobile terminal), the replacement of basemodel Xception, ShuffleNet; if you want to pursue precision (e.g., the AP paper brush, brush play list), basemodel then replaced ResNet-101 / ResNeXt / DPN.
3. train stage, usually directly into a pre-trained basemodel, fine-tune a, twenty epoch can be in their own data sets.