Jürgen Schmidhuber who is known to be given to the memory of artificial intelligence, neural networks recursive father, from 2004 to 2009, served as professor of cognition and robotics fields of the University of Munich, since 1995 as head of the Swiss AI Lab IDSIA people. Every year between 2009-2012, his team has won eight international matches pattern recognition and machine learning. Today Jürgen Schmidhuber Nnaisense founded the company.
1960 - 2013 time line depth study highlights
[A] 1962 Year: Implications Neurobiology from simple and complex cells of
Hubel and Wiesel describes a simple cells in the visual cortex and complex cells [18], it was later revealed the depth of artificial neural network framework, which is still used in some modern award-winning depth learning system.
[A0] 1965: The first generation of deep learning system
Ivakhnenko and Lapa [71] published the first paragraph for the general supervision of the former deep-fed Multilayer Perceptron (supervised deep feedforward multilayer perceptrons) and effective learning algorithm. In 1971 an article by describing a "data processing method group (Group Method of Data Handling)" Training for the network layer depth, still very popular in the new millennium. Consider a set of input training vectors corresponding to the target output vector, and the layer is gradually increased by regression analysis of training, followed by a help separate validation set of improvements, is the excess of the phase-out unit is used. The total number of units of each layer and the acquisition layer may be associated with the problem environment.
About [A1] 1970 ± 10 years: backpropagation
Complex multi-stage nonlinear error function differentiable, NN-related systems and their gradients from at least the early 1960 began to discuss the like [56-58,64-66]. In such systems can be at a gradient descent dynamic programming style (dynamic programming style) [67] to iterate the old chain rule [68, 69] (but with the chain rule simplified as compared with the down [57b] ). However, efficient error back-propagation (BP) on any, may be sparse, apparently using a similar network NN is the very first time in 1970 Linnainmaa [60-61] proposed. This is considered to be the reverse mode automatic differentiation, activation value (Activation) equal to the cost of the forward reverse differential propagation calculated value (cost) in nature. See Early FORTRAN codes [60]. Compare [62,29c] and some of the discussion about NN [29], and 1981 Werbos [29a, 29b] BP efficient algorithm of a specific NN. Comparison [30,31,59] and the recurrent neural network processing sequence is summarized as [32-34,37-39], see Nature gradient (naturalgradients). By 2013, BP also continues to be an important deep learning algorithms.
[A2] 1979 year: depth of the new learner (Deep Neocognitron), weight sharing and convolution
Fukushima depth perception new frame [19a, 19, 40] integrated view neurophysiology [A, 18] and the introduction of a convolutional neural layer sharing weights, and winner-takes-layer (winner-take-all layers ). It is similar to feedforward pure oversight modern award-winning depth gradient-based learning system [A11-A12] (but it uses local unsupervised learning rules).
[A3] 1987 year: automatic coding frame
Ballard published his ideas on non-supervised automatic encoder [35], which is associated with feed-forward depth learning system unsupervised pre-training based after 2000, such as [15, A8]. Surveys [36] and a certain relationship RAAMs [52].
[A4] 1989 Nian: CNN's back-propagation algorithm
Backpropagation LeCun et applications [16, 16a] Fukushima weight to weight of a convolutional neural layer [A2, 19a, 19, 16] share. The combination is an important part of many modern advantage in the competition for feed-forward visual depth learning system.
[A5] 1991 year: deep learning fundamental problem
20 In the early 1990s, experiments show that the depth of the former feed-forward or a recursive network is difficult to reverse the spread of training [A1]. Hochreiter my students to discover and analyze the reasons, because of the sudden disappearance of the gradient or gradient expansion (exploding) caused deep learning fundamental problem [3]. Comparison [4].
[A6] 1991 year: Recurrent Neural Networks depth frame
My first system recursive depth (mentioned above) [1,2] by a deep stack RNN pretraining (a deep RNN stack pre-trained in unsupervised fashion) in unsupervised situations, partially overcome the fundamental problem [ A5], thereby accelerating the later supervised learning. This is effective in depth learning system after 2000, and also the first time a nerve hierarchical memory model, is the first "deep learning system."
[A7] 1997: supervised depth learning system (LSTM)
Short and long term memory artificial neural network (LSTM RNN) became the first pure supervised depth learning systems, such as [5-10,12, A9]. LSTM RNN can find a lot of answers before learning can not solve the problem.
[A8] 2006: Convinced network (DeepBelief Network) / CNN results
Hinton and Salakhutdinov published articles mainly focus on unsupervised pre-training feed-forward NN to accelerate subsequent supervised learning (compare [A6]). This helps arouse people's interest about the depth of artificial network (keyword: Restricted Boltzmann Machine, convinced the network). In the same year, by using a training model deformation (training pattern deformations) [42, 43], Ranzato and others supervised BP training [A1, A4] of CNN [A2, A4] a record in MNIST handwriting digital image data sets benchmark record.
[A9] 2009 year: deep learning won the first contest
Deep learning won the first official international competition pattern recognition (there is a secret test set): LSTM RNN while performing segmentation and recognition [10, 11], there are some 2009 ICDAR victory handwriting contest link in [A7].
[A10] 2010 year: ordinary back-propagation algorithm on GPUs produced excellent results
But other aspects of the depth - no unsupervised pre-training, but there is no convolution training mode distortion - very standard neural network (NN) set a new record MNIST [17], a fast GPU by implementing the method of [17] . (A year later, the first on a human level performance MNIST system to produce --MCMPCNN [22, A11]).
[A11] 2011 Nian: MPCNN-- on the first super-human performance GPU visual pattern recognition
Ciresan, who introduced the GPU-based supervised the largest pool of CNN (convolution network) [21], today most (if not all) to gain advantage in the competition in depth using neural networks. By using a deep and extensive multi-column (Multi-Column, MC) GPU-MPCNN, deep learning system in visual pattern recognition (on the secret test set) the first time over the human performance [25,25a-c] ( twice as well as human performance, better three times than the nearest competition artificial neural network, six times better than the best non-neural method). Deep and wide multi-column (Multi-Column, MC) GPU-MPCNN is the current gold standard depth before feed-forward neural network, is now used in many applications.
[A12] 2012 Year: a first race in the object recognition and image segmentation victory
Image scanning [28,28a] GPU-MPCNN [21, A11] depth study to become a visual object detection system first race on a large picture of the winning (and only identify or classify opposite): 2012 ICPR mitosis detection contest . A popular in the computer vision community MC [A11] GPU-MPCNN variant model, setting a record in ImageNet classification benchmarks. Depth learning system for the first time in a pure picture contest (ISBI 2012) to win the division (the picture is a scan GPU-MPCNN) [53,53a, 53b].
[A13] 2013: more competition and benchmark record
LSTM create a new record TIMIT phoneme recognition [12]. Use the depth of GPU-MCMPCNN desktop machine ICDAR Chinese handwriting recognition benchmark (over 3700 categories) in a new record (almost Human Performance) [45a]. GPU-MPCNN [54-54b] won MICCAI2013 mitosis recognition awards Challenge. GPU-MPCNN [21] also helped to achieve in ImageNet PASCAL classification and object recognition [54e] in a new best mark in [26a]. More competition cases referred to the page GH Swiss AI Lab IDSIA and the University of Toronto.
END
Bi-mao wonderful classroom courses recommended:
1.Cloudera data analysis courses;
2.Spark Hadoop development and training;
3. Big Data machine learning recommendation systems;
4.Python data analysis and machine learning practical;
For more details, please look at our public number: Bi Mao data - Course Products - Bi-mao classroom
Register now interactive learning was massive currency, a large number of quality courses free delivery!