A Survey on Neural Architecture Search
Author: Martin Wistuba , Ambrish Rawat , Tejaswini Pedapati, three researchers have worked for IBM
Abstract
The growing interest in both the automation of machine learning and deep learning has inevitably led to the development of a wide variety of automated methods for neural architecture search.
Interest AutoML and AutoDL caused widespread rise of automated NAS approach.
The choice of the network architecture has proven to be critical, and many advances in deep learning spring from its immediate improvements.
Select the network architecture just important to enhance the depth of learning from many progressive architectural design.
However, deep learning techniques are computationally intensive and their application requires a high level of domain knowledge.
However, the depth of learning technology-intensive computing needs, the application process also requires a high level of knowledge areas
Therefore, even partial automation of this process helps to make deep learning more accessible to both researchers and practitioners .
Thus, even in part, researchers and related automation can make it easier for technicians to use deep learning.
With this survey, we provide a formalism which unifies and categorizes the landscape of existing methods along with a detailed analysis that compares and contrasts the different approaches.
Provides a paradigm for the unified classification and review by a conventional method, article, detail analysis and comparison of different methods.
We achieve this via a comprehensive discussion of the commonly adopted architecture search spaces and architecture optimization algorithms based on principles of reinforcement learning and evolutionary algorithms along with approaches that incorporate surrogate and one-shot models.
The main method of review is used in common comprehensive discussion of architecture search space and architecture optimization algorithm (reinforcement learning, evolutionary algorithms and methods include a proxy and a learning model).
Additionally, we address the new research directions which include constrained and multi-objective architecture search as well as automated data augmentation, optimizer and activation function search.
In addition, this paper presents some new research directions, including constrained multi-objective structure of the search, automatic data enhancement, optimization and activation function search.
Keywords: Neural Architecture Search, Automation of Machine Learning, Deep Learning, Reinforcement Learning, Evolutionary Algorithms, Constrained Optimization, MultiObjective Optimization
Introduction
Deep learning methods are very successful in solving tasks in machine translation, image and speech recognition. This success is often attributed to their ability to automatically extract features from unstructured data such as audio, image and text. We are currently witnessing this paradigm shift from the laborious job of manual feature engineering for unstructured data to engineering network components and architectures for deep learning methods. While architecture modifications do result in significant gains in the performance of deep learning methods, the search for suitable architectures is in itself a time-consuming, arduous and error-prone task. Within the last two years there has been an insurgence in research efforts by the machine learning community that seeks to automate this search process.
With its deep learning the ability to automatically extract features in the processing of unstructured data? Aspects highlight the advantages of restructuring to improve network performance for a significant effect, but the design suitable structure is a time-consuming, error-prone process. Over the past two years, the machine learning community opened to start the study of innovative, that is trying to make the search process automation network architecture.
On a high level, this automation is cast as a search problem over a set of decisions that define the different components of a neural network.The set of feasible solutions to these decisions implicitly defines the search space and the search algorithm is defined by the optimizer .
abstract view, this automated process can be regarded as a collection of search problems in decisions neural network components are defined in a set of. Feasible solution to these decisions implicitly defines the search space, and the optimizer defines the search algorithm.
Arguably, the works by Zoph and Le (Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017) and Baker et al. (Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures using reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017) mark the beginning of these efforts where their works demonstrated that good architectures can be discovered with the use of reinforcement learning algorithms. Shortly thereafter, Real et al. (Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2902– 2911) showed that similar results could also be achieved by the hitherto (迄今为止) well studied approaches in neuroevolution (Dario Floreano, Peter Du¨rr, and Claudio Mattiussi. Neuroevolution: from architectures to learning. Evolutionary Intelligence, 1(1):47–62, 2008.).
It can be said, NASNet early NAS to Bowen Baker, who made MetaQNN and Barret Zoph et al is representative. But then Real, who found neuroevolution evolutionary methods used to study the long-Floreano proposed reinforcement learning can be achieved with similar results.
However, both these search approaches consumed hundreds of GPU hours in their respective computations. Consequently, many of the subsequent works focused on methods that reduce this computational burden. The successful algorithms along this line of research leverage from the principle of reusing the learned model parameters, with the works of Cai et al. (Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 2787–2794, 2018a. URL https://www.aaai.org/ocs/index.php / Export / AAAI18 / paper / view / 16755.
Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path-level network transformation for efficient architecture search. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm¨assan, Stockholm, Sweden, July 10-15, 2018, pages 677–686, 2018b. URL http://proceedings.mlr.press/v80/ cai18a.html.
Han Cai, Ligeng Zhu, and Song Han ProxylessNAS:.. Direct neural architecture search on target task and hardware In Proceedings of the International Conference on Learning Representations, ICLR 2019, New Orleans, Louisiana, USA, 2019. URL https: // openreview .net / forum? id = HylVB3AqYm. ) and Pham et al. (Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research, pages 4095-4104) being the notable mentions.
However, the above methods are very GPU computing resource consumption, and thus a lot of work has focused on how to reduce the subsequent computational burden. During the search for the right value of reuse is more successful algorithm, a study along this line of thought has Cai and Pham et al work.
The design of the search space forms a key component of neural architecture search. In addition to speeding up the search process, this influences the duration of the search and the quality of the solution. In the earlier works on neural architecture search, the spaces were designed to primarily search for chain-structured architectures. However, with branched handcrafted architectures surpassing the classical networks in terms of performance, appropriate search spaces were proposed shortly after the initial publications (Zoph et al., 2018) and these have since become norm a in this field.
search space as a NAS essential elements, in addition to speed up the search process the outer (search speed), but also affects the duration (search period) and a mass (search results) searches the resultant structure. Early NAS research work, usually selected chain structure of the search space. However, as with the branch structure Zoph, who proposed gradually surpassed the performance of the classic network structure, the research community made many appropriate search space, and this structure with branches gradually called industry norms.
In parallel to these developments, researchers have broadened the horizons of neural architecture search to incorporate objectives that go beyond reducing the search time and generalization error of the found architectures. Methods that simultaneously handle multiple objective functions have become relevant. Notable works include methods that attempt to limit the number of model parameters or the like, for efficient deployment on mobile devices (Tan et al., 2018; Kim et al., 2017). Furthermore, the developed techniques for architecture search have been extended for advanced automation of other related components of deep learning. For instance, the search for activation functions (Ramachandran et al., 2018) or suitable data augmentation (Cubuk et al., 2018a).
These simultaneous development of many researchers broadening the scope of NAS and seek to optimize their search beyond the target time and generalization error, multi-objective optimization objective research methods thus simultaneously process becomes very meaningful. Representative work in this area include Tan and Kim et al achievements in terms of quantity and movement restrictions end deployment model parameters. Moreover, technology is also widely used in architecture search depth study optimization learning system other components, including search activation function studies and other aspects of the automatic data enhancement.
Currently, the automation of deep learning in the form of neural architecture search is one of the fastest developing areas of machine learning. With new papers emerging on arXiv.org each week and major conferences publishing a handful of interesting work, it is easy to lose track. With this survey, we provide a formalism which unifies the landscape of existing methods. This formalism allows us to critically examine the different approaches and understand the benefits of different components that contribute to the design and success of neural architecture search. Along the way, we also highlight some popular misconceptions pitfalls in the current trends of architecture search. We supplement our criticism with suitable experiments.
Currently, in order to search for the neural architecture as the representative of the depth of the automatic machine learning community learning is one of the fastest growing areas of research, every week the birth of a lot of interesting work, which is very likely to make people disoriented. By this review, we provide a paradigm for the unified direction of the existing sub-study to help us understand the various methods of NAS design criteria of success. In the process, we also highlighted the current schema search trends in some common misconceptions trap. We use the appropriate test to supplement our criticism.
Our review is divided into several sections. In Section 2, we discuss various architecture search spaces that have been proposed over time. We use Section 3 to formally define the problem of architecture search. Then we identify four typical types of optimization methods: reinforcement learning, evolutionary algorithms, surrogate model-based optimization, and one-shot architecture search. We define these optimization procedures and associate them to existing work and discuss it. Section 6 highlights the architecture search, considering constraints, multiple objective functions and model compression techniques. Alternate approaches that are motivated from the use of transfer learning are discussed in Section 5. Similarly, the class of methods that use early termination to fasten the search process are detailed in Section 4. Finally, the influence of search procedures on related areas is discussed in Section 7. The discussion is supported with extensive illustrations to elucidate （阐明）the different methods under a common formalism and relevant experimentation that examines the different aspects of neural architecture search methods.
This review is divided into seven parts unfold, Section II discusses different architectures search space, the third defines the architecture of the search problem and determine the four categories of typical optimization party specific analysis, namely reinforcement learning, evolutionary algorithms, the model agent and a one-shot search. After the optimization algorithm classification, we will contact each type of algorithm to specific articles were discussed. Similarly, in terms of accelerating the search, in the fourth quarter to take advantage of early stopping method technology to accelerate the search were discussed, Section V presents accelerate the migration method based learning. Section VI focuses on constraints to consider, as well as multi-objective model compression related search technology architecture. Section VII discusses the research progress in other areas related to the NAS, and by the thought of separation of variables studied different aspects of search methods of neural structures. Section VIII of technological developments and their applications NAS forecasted.
