Week 9 study report
1. Learning this week
1. Learned the micro-professional course "Deep Learning Engineer" by Mr. Wu Enda
Part 1: Neural Networks and Deep Learning
Week 4: Deep Neural Networks
Part 2: Improving Deep Neural Networks: Hyperparameter Tuning, Regularization, and Optimization
Week 1: The Practical Aspects of Deep Learning
2. Read the series of articles on the introduction of deep learning in Alibaba Cloud Yunqi Community
Chapter 8: Explains the Backpropagation (BP) algorithm in detail
Chapter 9: Mainly reviewed the development history of convolutional neural networks
Chapter 10: A brief introduction to convolutional neural networks and the application of convolution in image processing
Chapter 11: Topology of Convolutional Neural Networks
Chapter 12: The Rest of Convolutional Neural Networks: Pooling, Activation, and Fully Connected Layers
Chapter 13: Recurrent Neural Networks (RNNs)
3. Read a paper
Advancing state-of-the-artimage recognition with deep learning on hashtags
The study notes
The BP algorithm is actually not just a reverse algorithm, but a bidirectional algorithm.
Two major steps: (1) Forward propagation of signals and output classification information;
(2) Backpropagating the error and adjusting the network weights. If the intended purpose is not achieved, go back to (1) and (2).
The BP algorithm is insufficient, and there is a phenomenon of "Gradient Diffusion". The root of this is that for non-convex functions, once the gradient disappears, it has no guiding significance, resulting in it may be limited to local optima. Moreover, the phenomenon of "gradient diffusion" will become more and more serious as the number of network layers increases. As the gradient is reduced layer by layer, its effect on the adjustment of network weights is getting smaller and smaller, so the BP algorithm is mostly used in shallow network structures (usually less than or equal to 3), which limits the data representation ability of the BP algorithm , which limits the performance upper limit of BP.
Knowing that the main features of traditional CNN (Convolutional Neural Network) are "local connection", "weight sharing" and "local translation invariance", where "weight sharing" means "computation sharing", which saves a lot of computational overhead . RNN is different, it achieves "computation sharing" by implementing "advective transplantation" of parameters as the depth of "time" deepens.
The output of the RNN network is not only related to the current input, but also to the past output. Due to the utilization of historical information, when the task involves timing or context (such as speech recognition, natural language processing, etc.), RNN will perform much better than other artificial neural networks (such as CNN). The "depth" in RNN is different from the traditional deep neural network, which mainly refers to the depth in time and space (such as the number of hidden layers in the network).
3. Study plan for next week
1. Continue to study the micro-professional course of Mr. Wu Enda
2. After reading the series of articles about the introduction of deep learning in Yunqi Community
3. Continue to read the paper
4. Find a small demo online and train a model