Only high school mathematics can find algorithms? How powerful is Google's open source AutoML-Zero

Translator | Liu Chang

Source | AI Technology Base Camp (ID: rgznai100)

Machine learning research has made progress in many aspects, including model structure and optimization methods. There has also been significant progress in the work of automating such research (called AutoML). This progress is mainly focused on the architecture of neural networks, which currently rely on experts to design complex layers to build blocks (or similarly restricted search spaces).

The goal of this article is to prove that AutoML can go further. It is now possible to use only basic mathematical operations as building blocks to automatically discover complete machine learning algorithms.

This article introduces a new framework to prove this, the framework uses a more general search space, which can significantly reduce the subjective design willingness. Despite the large search space, evolutionary search can still find two-layer neural networks trained using back propagation. Then, these simple neural networks can be surpassed in some tasks. Even if these networks use the current top-level optimization algorithms, such as normalized gradients and weight averages.

In addition, this search can adapt the algorithm to different types of tasks: for example, when there is little available data, a dropout-like effect will occur. We believe that the initial success of discovering machine learning algorithms from scratch has indicated a very promising new direction for this research field.

introduction

In recent years, neural networks have achieved excellent performance on many critical tasks. The length and difficulty of machine learning research has spawned a new field called AutoML, which is to advance machine learning technology by spending machine computing time instead of human research time. This work has achieved fruitful results, but so far, the current research has relied heavily on the search space designed by humans. A common example is neural network architecture search, which uses complex layers designed by experts as building blocks and follows the rules of back propagation to limit the search space.

Similarly, other AutoML studies have also found ways to limit the search space to a single algorithm, such as the learning rules used during backpropagation, the LSTM gating structure, or data augmentation; in these studies, all other algorithms Still designed by hand. This method can save calculation time, but has two disadvantages. First of all, the artificially designed block structure will bias the search results to artificially designed algorithms, which may reduce the innovation ability of AutoML. Fewer options also limit innovation, because it is impossible to find content that cannot be searched. Second, the constrained search space needs to be carefully combined, which puts a new burden on researchers and violates the so-called goal of saving human time.

In order to solve this problem, this paper proposes a method for automatically searching all machine learning algorithms using only a few restrictions and simple mathematical operation modules. We call this method AutoML-Zero, and its purpose is to learn with the least human participation.

In other words, AutoML-Zero aims to search the model of fine-grained space, optimization process, initialization and other operations at the same time, thereby reducing the workload of manual design and even discovering non-neural network algorithms. To prove that this is feasible today, this article proposes a preliminary solution to this challenge.

The versatility of AutoML-Zero search space makes it more difficult than the existing AutoML algorithm corresponding to space search. The existing AutoML search space has built a dense and good solution, so the search method itself is no longer emphasized. For example, comparing in the same search space, it is found that leading technologies are usually only slightly better than simple random search (RS).

AutoML-Zero is different: because the search space is wider, the end result becomes very sparse. Our proposed framework expresses machine learning algorithms as computer programs containing three component functions that can predict and learn one sample at a time. The instructions in these functions assign basic mathematical operations to memory. The operations and memory addresses used by each instruction are free parameters in the search space, as is the size of the component function.

Overall, the contributions of this article are:

AutoML-Zero, which can automatically search the ML algorithm from the beginning with minimum manual participation;
New framework with source code and search space combined only with basic mathematical operations;
Detailed experimental results show the potential of using evolutionary search algorithms to discover ML algorithms.

Code address:

https://github.com/google-research/google-research/tree/master/automl_zero#automl-zero

method

The AutoML-zero method can be divided into two parts, one is the search space, and the other is the search method.

Search space

The author represents the algorithm as a computer program that functions on smaller virtual memory with separate address spaces for scalar, vector, and matrix variables. All of these are floating point numbers and share the dimensions of the task input feature map.

The author represents the program as a series of instructions. Each instruction has an operation to determine its function. In order to avoid the bias of the selection operation, this article uses a simple standard: it needs to be determined by high-level learning. The author purposely excluded machine learning concepts, matrix factorization and derivatives.

Inspired by supervised learning, the author expressed the algorithm as a program with three component functions, Setup / Predict / Learn. As shown below:

The evolution process in the figure below illustrates the use of the above functions. In the figure below, the two for loops implement the training and verification stages, and for simplicity, process one sample at a time. During the training phase, "prediction" and "learning" are alternately performed.

Search method

The search experiment must discover the machine learning algorithm by modifying the instructions in the component function. This article uses the regularized evolutionary search method because it is very simple and has recently achieved good results in architecture search. This method is shown in the figure below.

Mutant progeny that produced by the parent must be tailored to the search space; used herein, with the three types of operation unit selected: (i) a random position in the insert a random component in the function instruction or delete instruction, (ii) the Randomize all instructions in the component function, or (iii) modify one of the parameters of the instruction by replacing the instruction with a random selection. As shown below.

experiment

In the following experimental part, this article will answer the following three questions: "How difficult is it to search the AutoML-Zero space?", "Can the framework of this article be used to find a reasonable algorithm with minimal manual input?", And "Can I find different algorithms by changing the type of tasks used in the search experiment?"

1. Find a simple neural network in the search space

The following figure summarizes the analysis results of the four task types: found a complete algorithm / method of learning only linear / affine regression data. The AutoML-Zero search space is universal, but it comes at a price: even for simple tasks, good algorithms are sparse. As the task becomes more and more difficult, the solution becomes more and more sparse, and its performance is much better than RS.

2. Search with minimal manual input

Through searching, we found linear regression, two-layer neural network with back propagation, and even a baseline algorithm with a complexity that exceeds manual design. The figure above shows an example in our experiment, and we can see how the evolutionary algorithm solves the binary classification task step by step. The first is a linear model, without any optimization methods, and then gradually discovered SGD for optimization, then began to add a random learning rate, and then found the ReLU activation function, random weight initialization, gradient normalization, etc., more and more Network structure and optimization method close to manual design.

3. Discover the universality of the algorithm

In this section, the author will search for three different task types to demonstrate the broader applicability of this method. Each task type has its own challenges (for example, "too little data"). We will show the process of evolutionary adaptation algorithms to meet the challenges. Since we have obtained a reasonable model from scratch, now we only need to initialize the population using the effective neural network in the figure below, which can save time.

to sum up

In this article, the author proposes an ambitious goal for AutoML: to automatically discover the entire ML algorithm from basic operations, by reducing the preferences people bring in the search space, hoping this will eventually generate new ML content.

This paper builds a new framework for expressing ML algorithm to prove the potential of this research direction. The algorithm expresses ML algorithm as a computer program composed of three component functions (Setup, Predict, Learn). Starting from the empty component function, using only basic mathematical operations, the algorithm in this paper gradually evolved into linear regression, neural network, gradient descent, weight average, normalized gradient, etc. These results show that the algorithm is very promising, but there is still much work to be done.

Thesis address:

https://arxiv.org/abs/2003.03384

【END】