Summary:
Under normal circumstances, we will design the appropriate convolution neural network based on the current hardware resource, if the resource upgrade, the model structure can be enlarged for better accuracy. We systematically study the scaling model and verify the balance between network depth, width, and resolution in order to get better performance. Based on this idea, we propose a new scaling method: the use of complex scaling factor to unify all the dimensions of the model, the highest precision for maximum efficiency. Composite factor has: w convolution kernel size, d neural network depth, r resolution size. In the previous MobileNet and ResNet has demonstrated the efficiency of this method of scaling.
Search neural architecture designed a backbone network, and will enlarge the model to obtain a series of models, which we call Efficient, its accuracy and efficiency are better than all previous convolution network. In particular, our EfficientNet-B7 ImageNet obtained in the most advanced of the top-1 84.4% 97.1%, accuracy and precision of the top-5, as well as reduced 8.4 times greater than the size of the network, preferably before convolution, speed 6.1 times. Our EfficientNets also be a good migration, and to achieve the most advanced precision --CIFAR-100 (91.7%), Flowers (98.8%), the other three sets of data transfer learning.
Google latest paper: EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks ICML 2019
The appropriate source Address: EfficientNet
1, different network adjustment method: receptive field increases, the depth of the network increases, the resolution increases, as shown below:
Wherein, (A) is a baseline network, as may be appreciated that a small network; (b) to increase the receptive field of network expansion mode; (c) to increase the depth d of the extension network mode network; (d) to increase resolution extension rate r way network; (e) for the extension mode mixing parameters set forth herein; model scaled efficiency depends heavily on the baseline network, for further study, we used a network architecture developed new search network baseline, it was then scaled to obtain a series of models, called EfficientNets.
2, the relationship between test results, accuracy and efficiency at w, r, d curve in each case
Convolution general mathematical model is as follows:
H, W is the convolution kernel size, C is the number of channels, X is an input Tensor;
Determining a composite factor into the optimization problem as follows:
Adjust d, w, r so as to satisfy all the memory Memory and floating point number is smaller than the threshold requirement;
To achieve this goal, the paper proposes the following method:
For this method, we can look at two steps is determined d, w, r parameters:
First step may be adjusted by determining the best network baseline , then this parameter with the reference network to expand or enlarge a large network, so you can make large networks with high accuracy and efficiency. Similarly, we can extend the baseline network to other networks, using the same method to enlarge;
Baseline model using a mobile inverted bottleneck convolution (MBConv), similar to MobileNetV2 and MnasNet, but due to the increase FLOP budget, the larger model. Thus, researchers scale the baseline model, the model has been EfficientNets, its network diagram is as follows:
EfficientNet efficiency test:
对比EfficientNets和已有的CNN模型，EfficientNet 模型要比已有CNN模型准确率更高、效率更高，其参数量和 FLOPS 都下降了一个数量级，EfficientNet-B7 在 ImageNet 上获得了当前最优的 84.4% top-1 / 97.1% top-5 准确率，而且CPU 推断速度是 Gpipe 的 6.1 倍，但是模型大小方面，EfficientNet-B7却比其他模型要小得多，同时，还对比了ResNet-50，准确率也是胜出一筹(ResNet-50 76.3%，EfficientNet-B4 82.6%)。
3、相关工作：
ConvNet精度：自从AlexNet赢得2012的ImageNet竞赛，ConvNets随着变得更大获得了更高的精度，同时GoogleNet使用6.8M的参数获得了74.8%的top-1精度，SENet使145M参数获得了82.7%的top-1精度。最近，GPipe进一步提高了精度——使用了557M的参数获得了84.3%的top-1验证精度：它如此的大以至于需要将网络划分使用特定的并行化手段训练，然后将每一部分传递到一个不同的加速器。然而这些模型主要是为ImageNet设计，但是将其迁移到其他应用上时效果也很好。
ConvNet效率：深度网络的参数过多，模型压缩是一种通用的方法减小模型大小——平衡精度和效率。当移动网络变得无处不在时，我们也通常要手动设计高效的ConvNets，比如SqueezeNets、MobileNets、ShuffleNets。最近，神经网络结构搜索在设计高效的ConvNets变得越来越流行，并且通过广泛搜索网络宽度、深度、卷积核类型和大小得到了比手动设计的模型更高的精度。然而，目前还不清楚怎么将这些技术应用到更大的模型中，通常要更大的设计空间和更贵的微调成本，在本篇论文中，我们旨在研究效率设计更大规模的ConvNets，为了实现这个目标，我们采用模型缩放。
模型缩放：有很多的方法因不同的资源限制对ConvNet进行缩放：ResNet可以通过调整深度（缩小到ResNet-18，放大到ResNet-200），WideResNet和MobileNets可以通过对宽度（#channels）缩放。公认的是更大的输入图像尺寸有利于提高精度，同时需要更多的计算量。尽管之前的研究展示了网络深度和宽度对ConvNets的表现力很重要，它仍然是一个开放的问题来高效缩放ConvNet获得更好的效率和精度。我们的工作系统地研究了ConvNet对网络深度、宽度和分辨率这三个维度进行了缩放。
参考链接：
1、https://blog.csdn.net/Trent1985/article/details/91126085
2、https://blog.csdn.net/h__ang/article/details/92801712