On the afternoon of November 30th, the 2019 China Artificial Intelligence Industry Annual Conference released the " Report of Artificial Intelligence Development 2019 " (Report of Artificial Intelligence Development 2019).
Professor Tang Jie, deputy director of the Department of Computer Science and Technology of Tsinghua University, presided over the report release ceremony. Professor Guo Yuansheng, deputy director of the National Science and Technology Committee of the 1993 Central Committee of the Communist Party of China and vice chairman of the China Sensor and Internet of Things Industry Alliance, academician of the Japanese Academy of Engineering, and honorary vice chairman of the Chinese Society of Artificial Intelligence, The Associate Professor of Tokushima University in Japan and the guests present jointly unveiled the report.
Professor Tang Jie introduced the main content of the "2019 Artificial Intelligence Development Report" on behalf of the relevant units of the report. Original PPT
Professor Tang Jie made a speech
The report was jointly issued by the Tsinghua University-Chinese Academy of Engineering Knowledge Intelligence Joint Research Center and the Wu Wenjun Artificial Intelligence Science and Technology Award Selection Base of the Chinese Society of Artificial Intelligence. Beijing Zhipu Huazhang Technology Co., Ltd. provided technical support. The report seeks to comprehensively display the development status and trends of key areas of artificial intelligence in China and even the world, help the healthy development of the industry, and serve the country's strategic decision-making.
The report relies on the AMiner platform data resources and technology mining results to generate relevant data reports and charts, and invites Tsinghua University, Tongji University and other university experts to interpret core technologies and put forward opinions and suggestions, ensuring the scientificity and authority of the report to a certain extent.
The report covers 13 sub-areas of AI
The content of the report covers 13 sub-fields of artificial intelligence, including: machine learning, knowledge engineering, computer vision, natural language processing, speech recognition, computer graphics, multimedia technology, human-computer interaction, robotics, database technology, visualization, data mining, information Retrieval and recommendation.
The report presents two highlights
Professor Tang Jie introduced, "This report provides a detailed analysis of each sub-field of artificial intelligence, including basic concepts, development history, talent profile, interpretation of representative papers, and advances in cutting-edge technologies. Compared with the 2018 artificial intelligence development report , Has two bright spots. On the one hand, it is reflected in the "recent development of AI technology", and on the other hand, it is reflected in the "exploitation of talent network".
Professor Tang Jie made a speech
Regarding the highlight one "Recent development of AI technology", Professor Tang Jie gave a detailed explanation using "deep learning" as an example. Deep learning is the fastest growing branch of machine learning in the past 10 years. Due to its importance, three professors Geoffrey Hinton, Yann Lecun, and Yoshua Bengio won the 2018 Turing Award.
The development of deep learning models can be traced back to the Perceptron in 1958. In 1943, neural networks have already appeared in embryonic form (from NeuroScience). In 1958, Frank, a psychologist who studied cognition, invented the perceptron, which caused an upsurge at that time. Later, Marvin Minsky (artificial intelligence master) and Seymour Papert discovered the shortcomings of the perceptron: it was unable to handle non-linear problems such as XOR circuits, and there was a problem of insufficient computing power to handle large-scale neural networks, so the entire neural network research entered a period of stagnation. .
In the past 30 years, deep learning has achieved rapid development. The "2019 Artificial Intelligence Development Report" lists the four main contexts of deep learning. The top layer is the convolutional network, the middle layer is the unsupervised learning context, the next layer is the sequence depth model development context, and the bottom layer is the enhanced learning development context. . These four contexts fully demonstrate the recent development of "deep learning technology". (The following content is extracted from the second chapter of the report)
Important progress in deep learning models in recent years
The first development context (the light purple area in the figure above) is dominated by computer vision and convolutional networks. The development of this context can be traced back to Neocognitron proposed by Fukushima in 1979. The research gives the idea of convolution and pooling. Back propagation training MLP proposed by Hinton in 1986 (there were several similar studies before), which solved the problem that perceptrons cannot handle nonlinear learning. In 1998, researchers led by Yann LeCun implemented a seven-layer convolutional neural network LeNet-5 to recognize handwritten digits. Nowadays, Yann LeCun's research is generally regarded as the source of convolutional networks, but in fact, due to the rapid rise of SVM at that time, these neural network methods have not attracted widespread attention.
The event that really made convolutional neural networks in the elegant hall was that in 2012, AlexNet (a well-designed CNN) of the Hinton group won the championship with a huge advantage on ImageNet, which sparked the upsurge of in-depth learning. ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? AlexNet adds ReLU, Dropout and other techniques on the basis of traditional CNN, and the network scale is larger. These techniques later proved to be very useful and became the standard configuration of convolutional neural networks and were widely developed, so new models such as VGG and GoogLenet appeared later. In 2016, He Yuming, a young computer vision scientist, added jump connections between levels and proposed the residual network ResNet. ResNet greatly increases the depth of the network, and the effect is greatly improved. One that continues to develop this idea is DenseNet proposed by Huang Gao in the CVPR Best Paper in recent years. Various models (Mask-RCNN, etc.) have appeared for specific tasks in the field of computer vision. In 2017, Hinton believed that backpropagation and traditional neural networks still have certain shortcomings, so he proposed Capsule Net. This model enhances interpretability, but the current effect on data sets such as CIFAR is mediocre. This idea still needs to be verified and developed.
The second developmental context (the light green area in the figure above) is dominated by generative models. The traditional generative model is to predict the joint probability distribution P(x, y). Generative models in machine learning methods have always occupied a very important position, but generative models based on neural networks have not attracted widespread attention. In 2006, Hinton designed a machine learning generative model based on a restricted Boltzmann machine (RBM, an energy physical model based on an undirected graph model proposed in the 1880s), and stacked it into Deep Belief Network, using the layer-by-layer greedy or wake-sleep method of training, the effect of the model was not so good at that time. But it is worth noting that it is based on the RBM model Hinton and others began to design the deep framework, so this can also be seen as a beginning of deep learning.
Auto-Encoder is also a model proposed by Hinton in the 1980s, and later reappeared on the stage with the improvement of computing power. Bengio et al. also proposed Denoise Auto-Encoder, which is mainly aimed at possible noise problems in the data. Max Welling (this is also a master of variational and probabilistic graph models) and others later used neural networks to train a graph model with a layer of hidden variables. Because of the use of variational inference, and finally looked a bit like Auto-Encoder, it was called It is Variational Auto-Encoder. In this model, the distribution of hidden variables can be sampled, and the samples can be directly generated through the decoder network behind. The Generative Adversarial Network (GAN) is a very popular model proposed in 2014. It is a generative model that uses discriminators and generators for adversarial training. This idea is very distinctive. The model directly uses neural network G implicitly. Model the probability distribution of the entire sample. Each run is equivalent to sampling from the distribution. Later, it caused a lot of follow-up research, including: DCGAN is a fairly good implementation of convolutional neural network, WGAN is the work of measuring the similarity between distributions by replacing the original JS divergence by Weierstrass distance, making training stable . PGGAN increases the network layer by layer to generate realistic faces.
The third developmental context (the orange-yellow area in the figure above) is the sequence model. The sequence model is not created because of deep learning, but related research has been done a long time ago. For example, the hidden Markov HMM in the directed graph model and the conditional random field model CRF in the undirected graph model are very successful sequences. model. Even in the neural network model, Hopfield Network was proposed in 1982, that is, the idea of adding a recurrent network to the neural network. In 1997, Jürgen Schmidhuber invented the Long-Short Term Memory (LSTM) model, which was a landmark work. Of course, what really got the attention of the sequential neural network model was the 2013 Hinton group's use of RNN for speech recognition, which was much higher than traditional methods. In terms of text analysis, Yoshua Bengio, another Turing Award winner, proposed a neural network-based language model when SVM was very popular (of course, machine learning was still the world of SVM and CRF), and then Google proposed word2vec (2013 ) There are also some back-propagation ideas, the most important thing is to give a very efficient implementation, thus triggering the upsurge of research in this area.
Later, the RNN-based seq2seq model gradually appeared in tasks such as machine translation. The semantic information of a sentence was compressed into a vector through an Encoder, and then the translation result of the sentence was obtained through the Decoder conversion output. Later, the method was extended to Combined with the attention mechanism (Attention), it also greatly expands the model's representation capabilities and actual effects. Later, everyone found that using the CNN model with characters as the unit also performed well in many language tasks, and consumed less time and space. Self-attention actually adopts a structure to consider the local and global information of the same sequence at the same time. Google has a well-known article "Attention is all you need" which pushes the attention-based sequence neural model to a climax. Of course, there is also another article on ACL in 2019 to cool down this research slightly.
The fourth developmental context (the pink area in the figure above) is reinforcement learning. The most famous in this field is Deep Mind. The Dr. David Silver shown in the picture is an executive who has been studying RL. Q-learning is a well-known traditional RL algorithm. Deep Q-learning replaces the original Q-value table with a neural network and does a brick-and-mortar task. Later, it was applied in many game scenes, and its results were published on Nature. Double Dueling has made some extensions to this idea, mainly in the weight update timing of Q-Learning. Other DeepMind works such as DDPG and A3C are also very famous. They are variants based on the combination of Policy Gradient and neural network. AlphaGo, which everyone is familiar with, actually uses both RL methods and traditional Monte Carlo search techniques. Deep Mind later proposed a new algorithm Alpha Zero that uses AlphaGo's framework but uses main learning to play different (board) games.
Professor Tang Jie said: "The report also shows the development hotspots of deep learning in the past year or two. For example, when Google Bert was released last year, it caused a sensation in the entire industry and academia, or it will affect the future of deep learning and even the entire machine learning. The report sorts out Bert's related research in detail, and sorts out both the latest and most classic research, so that readers can see the future from related research."
Trend analysis can be generated by AMiner
Highlight one is also reflected in the detailed knowledge map. Professor Tang Jie pointed out, “Each field has a rich knowledge graph structure, and the development context of the entire field can be seen from the knowledge graph. At the same time, through such a knowledge graph, multi-level analysis including topic analysis and hot topic analysis can be further developed. Trend analysis, trend insight, etc.".
Regarding the second highlight, " Taking all talents in one network", Professor Tang Jie made a simple analysis. The report studied the distribution of scholars in various fields in the world and my country through in-depth mining and analysis of papers from top artificial intelligence journals/conferences and related scholars in the past 10 years. At the same time, the report further analyzes the gender ratio of scholars in various fields, h-index distribution, and China’s cooperation in various fields. It maps the authors to various countries through statistics on the author’s unit information in Chinese-foreign cooperative papers, and then calculates the relationship between China and China. The situation of collaborative papers between countries.
Professor Tang Jie said, “We have also developed precise portraits of talents and super-large-scale knowledge graphs. Through data mining, we first find the authors, and conduct in-depth talent portraits of each author. Not only can we see the contact information and position of each scholar. , Unit information, as well as position changes, interest changes, etc. On the other hand, through accurate portraits of scholars, the global distribution map and domestic distribution map of experts in each field can be compared and analyzed at home and abroad, and even brain drain can be carried out Analysis, such as analyzing whether a country’s talent introduction and talent outflow are profitable or loss-making."
Finally, Professor Tang Jie introduced AMiner, the data support platform for report writing. The AMiner system has been in operation for more than a decade since it was launched in 2006. It is a knowledge-driven scientific and technological intelligence mining platform, containing more than 300 million papers and more than 100 million scientific research personnel data, which can provide information including expert discovery, intelligent recommendation, institutional evaluation, Various services such as talent map and technology trend analysis. Many functions of the AMiner platform have been applied by the Ministry of Science and Technology, the Ministry of Education, the Beijing Municipal Commission of Science and Technology, and the Natural Science Foundation of China. It is hoped that the AMiner platform will have more application prospects in the future.
Contribute to the development of smart industry
At present, my country has entered a period of rapid development of science and technology. As a rising star in the field of science and technology, artificial intelligence is highly valued by the country. Under the guidance of multi-level strategic planning, whether in academia or industry, my country has a good performance in the international peers of artificial intelligence. The development of artificial intelligence in China has entered the fast lane.
At this stage, high-end talents who can promote technological breakthroughs and creative applications play a vital role in the development of artificial intelligence. The "2019 Artificial Intelligence Development Report" released this time provides an in-depth interpretation of the hotspots and cutting-edge technologies in recent years through in-depth research methods, and presents the latest research results. While focusing on the current status of artificial intelligence development, it also makes technical aspects. The analysis provides an outlook on the future development direction of related fields, and provides an information window for readers to understand recent development trends, basic and applied research results in artificial intelligence related fields.
The report is a professional report that integrates rigorous, comprehensive, technical, and forward-looking. It has extremely high academic value and reference value. It is not only conducive to the advancement of artificial intelligence research and exploration in my country, but also serves as an important reference for the country to understand the development trend of artificial intelligence and implement artificial intelligence development strategies.
END
Tips: If you need a paper version, please scan the following small program code, pure public welfare, only the printing cost will be charged.
Previous reading:
Emotional Computing of Artificial Intelligence (No. 6, 2019)