2020-04-01 12:43:32
Fish from the bottom of the concave sheep thirteen non-Temple
qubit reports | Public number QbitAI
Great God (tour) style all their own, separate ways Tiguan neural network.
Or are familiar with the team, or the familiar signature, Facebook AI Lab, network design innovation challenges of the new paradigm.
Ah, familiar with Ross, familiar Ho Kai Ming, they bring a new - RegNet .
Not only with the current mainstream network design paradigm "contrary": simple, easy to understand the model, but also a high amount of calculation can hold live.
Also under similar conditions, but also better than the performance EfficientNet, the speed on the GPU also increases the 5-fold !
The new network design paradigm that combines manual design of network and neural network search (NAS) advantages:
And manual design networks, which are explanatory goal may be described generally simple network design principles and generalized in various settings.
Like another and NAS, you can use semi-automatic processes, to find easy to understand, build and generalization of a simple model.
There is no doubt in the paper also CVPR 2020.
PS: March 30 paper is published in arXiv, So we do not have to worry about an April Fool's joke ......
Three sets of experimental contrast, almost "Grand Slam"
RegNet in performance is so outstanding.
Experiments were performed on ImageNet dataset goal is very clear: the challenges of neural networks in a variety of environments .
Let's look at, with a popular public neural network of the mobile end of the comparison.
Recently, many network design work has focused on moving mechanisms (mobile regime, ~600MF).
Table RegNet is 600MF, the result of the comparison with these networks. As can be seen, whether it is based on manual design or NAS network, RegNe performed very well.
Kai-Ming Ho stressed group, using the basic model RegNet 100 epoch schedule (Schedule), in addition to attenuation of the weight, there is no use of any regularization.
Most mobile networks use a longer schedule, and a variety of enhancements, such as the depth of supervision, Cutout, DropPath and so on.
Next, is RegNet with standard baseline ResNet and ResNeXT comparison.
To be fair, researchers at the same training set, compare them, as shown below:
Overall, by optimizing the network structure, RegNet model in all the complexity of the indicators have been greatly improved.
The researchers also stressed that good RegNet model is also suitable for a wide range of computing environments, including low computing environment ResNet and ResNeXT are not comfortable with.
In the table (a) in accordance with the present comparative activation packet.
The researchers defined as the output activation tensor of all conv layer size, it would be like this GPU accelerated degree have a greater impact operation.
The researchers said that the significance of this set is very large, because the model training time is a bottleneck. In this possible future scenario autopilot, help improve inference time. The reasoning given a fixed time or training, RegNet very effective.
In the table in (b), shows a comparison of the packet according flops.
Finally, RegNet with EfficientNet comparison.
EfficientNet, it represents the most popular technique, comparing the results shown below:
It can be seen at the lower flops, EfficientNet there are advantages, but with the increase of flops, RegNetX and RegNetY gradually force.
In addition, HE Ming Chan team found, for EfficentNet, activation with a linear relationship flops; for RegNet, activation flops linear relationship to the square root.
This leads to slower GPU training and inference of EfficiententNet. RegNeTX-8000 and faster than Efficient entNet-B5 5 times, while having less error, as follows:
So performance, the next question is, exactly how RegNet Make?
To build the network design space
Here first introduce Radosavovic, who proposed network design space (network design spaces) concept.
The core idea is that the model can be sampled in the design space, resulting in the distribution model, and can be used in the classical statistical tools to analyze the design space.
In this study, Ho Kai Ming's team, the researchers propose, design a gradually simplified version unrestricted initial design space. This process, called design space design (design space design).
In each step of the design process, the input is the initial design space, the output is a more simple, compact models or better performance model.
Sampled by the model, and checking the error distribution, to characterize the quality of the design space.
For example, in the figure, from the initial design space A, application of two optimization steps to generate the design space B, and then C.
C⊆B⊆A, can be seen, then C from A to B, the error distribution gradually improved.
In other words, the purpose of each design step, are able to generate in order to find a simpler, more efficient model design principles.
The researchers designed the initial design space is AnyNet .
The basic design of the network is very simple: the trunk (2 steps of 3 × 3 convolution, 32 output channels) performs computationally intensive + body + the predicted output category network head (pooled average, then fully connected layer).
Internet body is composed of a series of stages, these stages operating at progressively lower resolution.
Except for the first block (steps convolution using 2) except that each stage comprises a series of identical blocks.
Although the overall structure is very simple, but the total number of network AnyNet design space that may exist in very large.
Most experiments using a standard block with a residual packet convolution bottleneck, researchers called X block, constructed on the basis of its design space AnyNet referred AnyNetX.
On AnyNetX, the researchers aimed to achieve four purposes:
- Simplify the design space structure
- Improve the design space interpretability
- To improve or maintain the quality of the design space
- To maintain the diversity of the design space model
Thus, the initial AnyNetX referred AnyNetXA, start "A → B → C → D → E" optimization process.
First of all, for all stages of the design space AnyNetXA, test the shared bottleneck rate (bottleneck ratio) bi = b, and the resulting design space becomes AnyNetXB.
Also, at the same settings, sample and 500 models from training AnyNetXB.
AnyNetXA and AnyNetXB in the average case and best-case scenario, EDF is almost unchanged. No loss of accuracy described coupling bi. And, AnyNetXB easier to analyze.
Then, from the start AnyNetXB, the width of the group using shared (shared group width) for all phases to obtain AnyNetXC.
As before, EDF hardly changed.
Then, the researchers tested the network in good and bad network AnyNetXC in a typical network structure.
They found that: the width of a good network is in the form of growth.
Then, they added the design principles wi + 1 ≥ wi, having this design space constraints referred AnyNetXD.
This greatly improves the EDF.
△左：AnyNetXD，右：AnyNetXE
For the best model, not just the stage width wi is increasing, researchers found that the depth di stage have the same trend, except for the last stage.
Then, after addition of di + 1 ≥ di constraints, results again improved. That AnyNetXE.
In further observation in AnyNetXE, leads to a central point of view of RegNet: width and depth of a good network is quantized linear function to explain.
From AnyNetXA to RegNetX, dimensions from 16 to 6 dimension dimension, size reduction of nearly 10 orders of magnitude.
As can be seen from the figure, RegNetX AnyNetX the model compared to the model, lower average error. And, RegNetX random search efficiency is much higher, about 32 searches the stochastic model can produce good model.
Design space generalization
At first, in order to improve efficiency, the researchers calculated the amount of low, low-epoch of training methods designed RegNet design space.
But their goal is not tied to a specific environment, but found that the general principles of network design.
Thus, they are higher FLOPS, higher epoch network 5 in steps, and various types of blocks, and comparing the RegNetX AnyNetXA, AnyNetXE.
In all cases, the design space have not seen fit phenomenon.
In other words, RegNet good generalization ability.
Finally, or conventionally to introduce this dream team AI research it.
Familiar faces, familiar team
Ross and Ho Kai Ming, this combination is very familiar.
This time the five authors, all from Facebook AI Research Institute.
A paper for, Ilija Radosavovic, undergraduate teaching assistant at Imperial College London, worked as an intern at Facebook.
Raj Prateek Kosaraju and Ross Girshick, Dr. graduated from the Georgia Institute of Technology and the University of Chicago, is a computer scientist FAIR visual direction.
Finally, author Piotr Dollar, graduated from the University of California, San Diego, is also worked FAIR.
Chinese people are most familiar with natural genius AI researcher Ho Kai and clear.
Ho Kai Ming and RegNet The team proposed Judging from the name, is also quite similar to his own masterpiece of the year --ResNet-- 2016 Nian CVPR Best Paper Award.
In addition, Kaiming Great God respectively in 2009 and 2017, won the Best Paper Award ICCV CVPR and so far there are still hard to newcomers.
(Continue to worship ing ......)
Interestingly, in this study, but also to do ResNet as a baseline comparison.
But not unexpectedly, from the research point of view in recent years, Ho Kai Ming also constantly break their own previous methods, research.
Kai Ming on the road beyond what is currently the fastest, Ho Kai Ming is still himself.
Ah, the god of joy, is so plain and low-key.
Our first film for the King, on how to evaluate RegNet leave you friends ~
Portal
Papers address: https: //arxiv.org/pdf/2003.13678.pdf
- Finish-
