Author | Anthony Alford Translator | PingchuanUsing synthetic data, Uber has improved its neural structure search (NAS) deep learning optimization process by 9 times.
Uber Artificial Intelligence Lab has developed an algorithm called Generative Teaching Network (GTN), which generates synthetic training data for neural networks, making neural network training faster than when using real data. Using these synthetic data, Uber has improved its neural structure search (NAS) deep learning optimization process by 9 times.
In a paper published on arXiv, the team described the system and a series of experiments. The motivation of GTN comes from the neural structure search (NAS) problem, which trains many different deep learning model structures and selects the best performing one for a set of test data. Although a typical method is to train each model through multiple iterations (or epochs) on a complete data set, this is time-consuming and expensive. However, the model can be trained on GTN synthetic data, which takes less time, and can estimate its true performance on real training data; this can quickly evaluate the model and shorten the search time. According to the researchers:
GTN-Neural Structure Search (GTN-NAS) is comparable to the current state-of-the-art NAS methods. These methods perform best and have a few orders of magnitude less computation than typical NAS methods.
Neural structure search is an active research area in Automatic Machine Learning (AutoML). NAS has a shortcoming, it requires training many deep learning models to determine which model performs best. The research focuses on efficiently exploring the search space, which means that the system will train fewer models. Uber's system generates a new data set, so that each model is iteratively trained fewer times, so that the system can test more models in the same amount of time.
The problem with training models through fewer iterations is that in the very early training stage, most models have poor performance, and multiple iterations are required to determine the true performance of the model. However, research shows that not all training samples are like this. Careful selection of input samples can speed up training. Uber's idea is to use meta-learning to generate training samples. Similar to the Generative Adversarial Network (GAN), Uber's GTN will train a generator neural network to generate training samples for the learner network. The learner evaluates based on the real test data, generates a "meta loss", and updates the generator with the gradient of the meta loss. Using this technology, Uber created a generator to generate samples used to train a computer vision (CV) system to recognize digits in the MNIST data set. The CV system only needs 32 training steps to achieve an accuracy of 98.9%. Uber said that in a similar experiment on the CIFAR10 dataset, they used synthetic data to predict model performance in 128 steps, while using real data requires 1,200 steps, which is a 9-fold increase in speed.
The co-author of the paper, Jeff Clune, tweeted a picture of synthetic image data generated by the system and described it as "alternative and unrealistic." He also said:
In the neural structure search, the data generated by GTN can completely replace the real data, which can greatly accelerate the speed of any NAS algorithm. So far, we have only shown random search-NAS (plus some fancy features), but we hope to see others try other more advanced NAS methods!