In many important applications such as autonomous driving, the data is dynamic in real time, and some unexpected situations may occur from time to time. In order to effectively deal with this problem, MIT researchers designed a new type of neural network inspired by biological neurons, and they also demonstrated the effectiveness of the neural network through theoretical proofs and experimental verifications. The relevant code has also been announced.
Researchers at the Massachusetts Institute of Technology (MIT) have developed a new type of neural network that can not only learn during the training phase, but also continuously adapt. They named this flexible algorithm "Liquid" network because it can change its underlying mathematical equations like "liquid" to continuously adapt to new input data. This progress can help decision-making tasks based on dynamically changing data, such as tasks involved in medical diagnosis and autonomous driving.
Paper address: https://arxiv.org/pdf/2006.04439.pdf
Code address: https://github.com/raminmh/liquid_time_constant_networks
"This road can lead to the future of robot control, natural language processing, video processing-any form of time series data processing." Ramin Hasani, the lead author of the study, said, "It has great potential."
This research paper is one of the selected papers of the AAAI 2021 conference.
Hasani said that in order to understand the world, time series data is not only ubiquitous, but also vital and indispensable. "The real world is all related to sequences. The way we perceive is the same-what you perceive is not an image, but a sequence of images." He said, "So, time series data actually creates our reality."
He pointed out that video processing, financial data and medical diagnostic applications all involve time series, and these applications are vital to our society. The changes in these ever-changing data streams are difficult to predict. However, if these data can be analyzed in real time and used to predict future behavior, then the development of technologies such as autonomous driving can be greatly promoted.
Researchers such as Hasani have designed a neural network that can adapt to changes in real-time world systems. The design of the neural network is inspired by the biological brain, and Hasani said that their design of this specific neural network is directly inspired by C. elegans. He said: "Its nervous system has only 302 neurons, but it can produce complex dynamics that exceed expectations."
By carefully observing the activation methods of Caenorhabditis elegans neurons and how they communicate with each other through electrical impulses, Hasani coded his neural network. In its equations for building neural networks, parameters can change over time based on the results of a set of nested differential equations.
Algorithm 1: Liquid time constant (LTC) cyclic neural network realized by the aggregated ordinary differential equation (ODE) solving algorithm, where θ is the parameter space and f can be any activation function.
Algorithm 2: Train LTC through back propagation over time (BPTT).
This flexibility is the key. After the training phase, the behavior of most neural networks is fixed, which means that they are difficult to adjust to changes in the input data stream. Hasani said that the fluidity of his Liquid network makes it more flexible to deal with unexpected or noisy data , such as torrential rain that obscures the camera view of self-driving cars. "In other words, it is more robust."
Hasani added that network flexibility has another big advantage: "It can also be explained better."
Hasani said that his Liquid network circumvents the intricacies common to other neural networks . "Just by changing the way neurons are represented, you can explore a certain level of complexity that cannot be explored in other ways." Hasani's method of change is to use differential equations. Thanks to this small number of neurons with high characterization capabilities, it is easier to snoop into the "black box" of the network's decision-making process and diagnose why the network has certain characteristics.
Hasani said: "The model itself is rich in expressiveness." This can help engineers understand and improve the performance of the Liquid network.
Figure 1: Expressiveness is measured by trajectory length. The trajectory hidden space of static deep networks will become more complicated as the input passes through the hidden layer.
Figure 2: LTC using different activation functions as a measure of expressiveness by trajectory length.
Hasani's network has achieved excellent performance in a series of tests. In tasks ranging from atmospheric chemistry to traffic pattern analysis, the newly proposed method outperforms other current best time series algorithms by several percentage points in predicting future values. In addition, due to the small size of the network, the computational cost in the test is also much lower. "Everyone is talking about expanding their network," Hasani said. "We want to shrink so that we can get fewer but richer nodes."
Time series forecast results.
Hasani plans to continue to improve the system and explore its industry applications. "Inspired by nature, we already have a more expressive neural network that has been proven. But the process has just begun." He said, "The obvious question is: how do we scale it? We think this type of network will Become a key component of the future intelligent system."
【Editor's Choice】
- 5 things you need to know about artificial intelligence application scenarios
- Re-learning the principles of computer composition-detailed computer bus
- Android 12 is trying to use machine learning to optimize gesture navigation
- Making the data center smarter: how does artificial intelligence work?
- Five major trends in artificial intelligence in 2021