转 给研一同学们的深度学习学习规划

转载于： https://blog.csdn.net/u014722627/article/details/64919741

 

DeepLearning Syllabus

——整理by xrj[[email protected]]

学习路线

1. 完成cs231n学习.
2. 熟悉python，并完成对应assignment.
3. 按照课程基础部分阅读相应基础paper.
4. 按兴趣阅读特定应用方向paper.
5. 学习某一种深度学习框架，并实现复现感兴趣方向一篇paper的实验结果.
6. 总结汇报.

先修理论知识（用到再看）：

微积分（复合导数）

概率论与数理统计

线性代数（高等代数）

工具平台（推荐）

1. Anaconda(python 2.7/3.5) —cs231n的assignment用2.7版本
   1. numpy（矩阵运算，数值计算库）
   2. matplotlib（专业绘图）
   3. pandas（数据清洗、切片预处理）
   4. sklearn（特征选择、传统ML）
   5. python-opencv
2. 深度学习框架：Tensorflow/Pytorch/mxnet/caffe(不推荐).
3. 离线浏览器文档查阅：devdocs.io （包含numpy、pandas等）

课程学习(CS231n)

PPT 视频 ：链接：http://pan.baidu.com/s/1qYe11Zu 密码：lijn

Assignmen1 :

1. k-Nearest Neighbor classifier
2. Training a Support Vector Machine
3. Implement a Softmax classifier
4. Two-Layer Neural Network

Assignmen2

1. Fully-connected Neural Network
2. Batch Normalization
3. Dropout
4. ConvNet on CIFAR-10

Assignmen3(选做)

1. Image Captioning with Vanilla RNNs
2. Image Captioning with LSTMs
3. Image Gradients: Saliency maps and Fooling Images
4. Image Generation: Classes, Inversion, DeepDream

Papers1

Understanding / Generalization / Transfer

• Distilling the knowledge in a neural network (2015), G. Hinton et al. [pdf]
• Deep neural networks are easily fooled: High confidence predictions for unrecognizable images (2015), A. Nguyen et al. [pdf]
• How transferable are features in deep neural networks? (2014), J. Yosinski et al. [pdf]
• CNN features off-the-Shelf: An astounding baseline for recognition (2014), A. Razavian et al. [pdf]
• Learning and transferring mid-Level image representations using convolutional neural networks (2014), M. Oquab et al. [pdf]
• Visualizing and understanding convolutional networks (2014), M. Zeiler and R. Fergus [pdf]
• Decaf: A deep convolutional activation feature for generic visual recognition (2014), J. Donahue et al. [pdf]

Optimization / Training Techniques

• Batch normalization: Accelerating deep network training by reducing internal covariate shift (2015), S. Loffe and C. Szegedy [pdf]
• Delving deep into rectifiers: Surpassing human-level performance on imagenet classification (2015), K. He et al. [pdf]
• Dropout: A simple way to prevent neural networks from overfitting (2014), N. Srivastava et al. [pdf]
• Adam: A method for stochastic optimization (2014), D. Kingma and J. Ba [pdf]
• Improving neural networks by preventing co-adaptation of feature detectors (2012), G. Hinton et al. [pdf]
• Random search for hyper-parameter optimization (2012) J. Bergstra and Y. Bengio [pdf]

Unsupervised / Generative Models

• Pixel recurrent neural networks (2016), A. Oord et al. [pdf]
• Improved techniques for training GANs (2016), T. Salimans et al. [pdf]
• Unsupervised representation learning with deep convolutional generative adversarial networks (2015), A. Radford et al. [pdf]
• DRAW: A recurrent neural network for image generation (2015), K. Gregor et al. [pdf]
• Generative adversarial nets (2014), I. Goodfellow et al. [pdf]
• Auto-encoding variational Bayes (2013), D. Kingma and M. Welling [pdf]
• Building high-level features using large scale unsupervised learning (2013), Q. Le et al. [pdf]

Convolutional Neural Network Models

• Rethinking the inception architecture for computer vision (2016), C. Szegedy et al. [pdf]
• Inception-v4, inception-resnet and the impact of residual connections on learning (2016), C. Szegedy et al. [pdf]
• Identity Mappings in Deep Residual Networks (2016), K. He et al. [pdf]
• Deep residual learning for image recognition (2016), K. He et al. [pdf]
• Going deeper with convolutions (2015), C. Szegedy et al. [pdf]
• Very deep convolutional networks for large-scale image recognition (2014), K. Simonyan and A. Zisserman [pdf]
• Spatial pyramid pooling in deep convolutional networks for visual recognition (2014), K. He et al. [pdf]
• Return of the devil in the details: delving deep into convolutional nets (2014), K. Chatfield et al. [pdf]
• OverFeat: Integrated recognition, localization and detection using convolutional networks (2013), P. Sermanet et al. [pdf]
• Maxout networks (2013), I. Goodfellow et al. [pdf]
• Network in network (2013), M. Lin et al. [pdf]
• ImageNet classification with deep convolutional neural networks (2012), A. Krizhevsky et al. [pdf]

Image: Segmentation / Object Detection

• You only look once: Unified, real-time object detection (2016), J. Redmon et al. [pdf]
• Region-based convolutional networks for accurate object detection and segmentation (2016), R. Girshick et al. [pdf]
• Fully convolutional networks for semantic segmentation (2015), J. Long et al. [pdf]
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (2015), S. Ren et al. [pdf]
• Fast R-CNN (2015), R. Girshick [pdf]
• Rich feature hierarchies for accurate object detection and semantic segmentation (2014), R. Girshick et al. [pdf]
• Semantic image segmentation with deep convolutional nets and fully connected CRFs, L. Chen et al. [pdf]
• Learning hierarchical features for scene labeling (2013), C. Farabet et al. [pdf]

Image / Video / Etc

• Image Super-Resolution Using Deep Convolutional Networks (2016), C. Dong et al. [pdf]
• A neural algorithm of artistic style (2015), L. Gatys et al. [pdf]
• Deep visual-semantic alignments for generating image descriptions (2015), A. Karpathy and L. Fei-Fei [pdf]
• Show, attend and tell: Neural image caption generation with visual attention (2015), K. Xu et al. [pdf]
• Show and tell: A neural image caption generator (2015), O. Vinyals et al. [pdf]
• Long-term recurrent convolutional networks for visual recognition and description (2015), J. Donahue et al. [pdf]
• VQA: Visual question answering (2015), S. Antol et al. [pdf]
• DeepFace: Closing the gap to human-level performance in face verification (2014), Y. Taigman et al. [pdf]:
• Large-scale video classification with convolutional neural networks (2014), A. Karpathy et al. [pdf]
• DeepPose: Human pose estimation via deep neural networks (2014), A. Toshev and C. Szegedy [pdf]
• Two-stream convolutional networks for action recognition in videos (2014), K. Simonyan et al. [pdf]
• 3D convolutional neural networks for human action recognition (2013), S. Ji et al. [pdf]

Recurrent Neural Network Models

• Conditional random fields as recurrent neural networks (2015), S. Zheng and S. Jayasumana. [pdf]
• Memory networks (2014), J. Weston et al. [pdf]
• Neural turing machines (2014), A. Graves et al. [pdf]
• Generating sequences with recurrent neural networks (2013), A. Graves. [pdf]

Natural Language Processing

• Neural Architectures for Named Entity Recognition (2016), G. Lample et al. [pdf]
• Exploring the limits of language modeling (2016), R. Jozefowicz et al. [pdf]
• Teaching machines to read and comprehend (2015), K. Hermann et al. [pdf]
• Effective approaches to attention-based neural machine translation (2015), M. Luong et al. [pdf]
• Neural machine translation by jointly learning to align and translate (2014), D. Bahdanau et al. [pdf]
• Sequence to sequence learning with neural networks (2014), I. Sutskever et al. [pdf]
• Learning phrase representations using RNN encoder-decoder for statistical machine translation (2014), K. Cho et al. [pdf]
• A convolutional neural network for modeling sentences (2014), N. Kalchbrenner et al. [pdf]
• Convolutional neural networks for sentence classification (2014), Y. Kim [pdf]
• Glove: Global vectors for word representation (2014), J. Pennington et al. [pdf]
• Distributed representations of sentences and documents (2014), Q. Le and T. Mikolov [pdf]
• Distributed representations of words and phrases and their compositionality (2013), T. Mikolov et al. [pdf]
• Efficient estimation of word representations in vector space (2013), T. Mikolov et al. [pdf]
• Recursive deep models for semantic compositionality over a sentiment treebank (2013), R. Socher et al. [pdf]

Speech / Other Domain

• End-to-end attention-based large vocabulary speech recognition (2016), D. Bahdanau et al. [pdf]
• Deep speech 2: End-to-end speech recognition in English and Mandarin (2015), D. Amodei et al. [pdf]
• Speech recognition with deep recurrent neural networks (2013), A. Graves [pdf]
• Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups (2012), G. Hinton et al. [pdf]
• Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition (2012) G. Dahl et al. [pdf]
• Acoustic modeling using deep belief networks (2012), A. Mohamed et al. [pdf]

Reinforcement Learning / Robotics

• End-to-end training of deep visuomotor policies (2016), S. Levine et al. [pdf]
• Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection (2016), S. Levine et al. [pdf]
• Asynchronous methods for deep reinforcement learning (2016), V. Mnih et al. [pdf]
• Deep Reinforcement Learning with Double Q-Learning (2016), H. Hasselt et al. [pdf]
• Mastering the game of Go with deep neural networks and tree search (2016), D. Silver et al. [pdf]
• Continuous control with deep reinforcement learning (2015), T. Lillicrap et al. [pdf]
• Human-level control through deep reinforcement learning (2015), V. Mnih et al. [pdf]
• Deep learning for detecting robotic grasps (2015), I. Lenz et al. [pdf]
• Playing atari with deep reinforcement learning (2013), V. Mnih et al. [pdf])

More Papers from 2016

• Layer Normalization (2016), J. Ba et al. [pdf]
• Learning to learn by gradient descent by gradient descent (2016), M. Andrychowicz et al. [pdf]
• Domain-adversarial training of neural networks (2016), Y. Ganin et al. [pdf]
• WaveNet: A Generative Model for Raw Audio (2016), A. Oord et al. [pdf] [web]
• Colorful image colorization (2016), R. Zhang et al. [pdf]
• Generative visual manipulation on the natural image manifold (2016), J. Zhu et al. [pdf]
• Texture networks: Feed-forward synthesis of textures and stylized images (2016), D Ulyanov et al. [pdf]
• SSD: Single shot multibox detector (2016), W. Liu et al. [pdf]
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 1MB model size (2016), F. Iandola et al. [pdf]
• Eie: Efficient inference engine on compressed deep neural network (2016), S. Han et al. [pdf]
• Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1 (2016), M. Courbariaux et al. [pdf]
• Dynamic memory networks for visual and textual question answering (2016), C. Xiong et al. [pdf]
• Stacked attention networks for image question answering (2016), Z. Yang et al. [pdf]
• Hybrid computing using a neural network with dynamic external memory (2016), A. Graves et al. [pdf]
• Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016), Y. Wu et al. [pdf]

---

1. https://github.com/terryum/awesome-deep-learning-papers ↩

DeepLearning Syllabus

——整理by xrj[[email protected]]

学习路线

1. 完成cs231n学习.
2. 熟悉python，并完成对应assignment.
3. 按照课程基础部分阅读相应基础paper.
4. 按兴趣阅读特定应用方向paper.
5. 学习某一种深度学习框架，并实现复现感兴趣方向一篇paper的实验结果.
6. 总结汇报.

先修理论知识（用到再看）：

微积分（复合导数）

概率论与数理统计

线性代数（高等代数）

工具平台（推荐）

1. Anaconda(python 2.7/3.5) —cs231n的assignment用2.7版本
   1. numpy（矩阵运算，数值计算库）
   2. matplotlib（专业绘图）
   3. pandas（数据清洗、切片预处理）
   4. sklearn（特征选择、传统ML）
   5. python-opencv
2. 深度学习框架：Tensorflow/Pytorch/mxnet/caffe(不推荐).
3. 离线浏览器文档查阅：devdocs.io （包含numpy、pandas等）

课程学习(CS231n)

PPT 视频 ：链接：http://pan.baidu.com/s/1qYe11Zu 密码：lijn

Assignmen1 :

1. k-Nearest Neighbor classifier
2. Training a Support Vector Machine
3. Implement a Softmax classifier
4. Two-Layer Neural Network

Assignmen2

1. Fully-connected Neural Network
2. Batch Normalization
3. Dropout
4. ConvNet on CIFAR-10

Assignmen3(选做)

1. Image Captioning with Vanilla RNNs
2. Image Captioning with LSTMs
3. Image Gradients: Saliency maps and Fooling Images
4. Image Generation: Classes, Inversion, DeepDream

Papers1

Understanding / Generalization / Transfer

• Distilling the knowledge in a neural network (2015), G. Hinton et al. [pdf]
• Deep neural networks are easily fooled: High confidence predictions for unrecognizable images (2015), A. Nguyen et al. [pdf]
• How transferable are features in deep neural networks? (2014), J. Yosinski et al. [pdf]
• CNN features off-the-Shelf: An astounding baseline for recognition (2014), A. Razavian et al. [pdf]
• Learning and transferring mid-Level image representations using convolutional neural networks (2014), M. Oquab et al. [pdf]
• Visualizing and understanding convolutional networks (2014), M. Zeiler and R. Fergus [pdf]
• Decaf: A deep convolutional activation feature for generic visual recognition (2014), J. Donahue et al. [pdf]

Optimization / Training Techniques

• Batch normalization: Accelerating deep network training by reducing internal covariate shift (2015), S. Loffe and C. Szegedy [pdf]
• Delving deep into rectifiers: Surpassing human-level performance on imagenet classification (2015), K. He et al. [pdf]
• Dropout: A simple way to prevent neural networks from overfitting (2014), N. Srivastava et al. [pdf]
• Adam: A method for stochastic optimization (2014), D. Kingma and J. Ba [pdf]
• Improving neural networks by preventing co-adaptation of feature detectors (2012), G. Hinton et al. [pdf]
• Random search for hyper-parameter optimization (2012) J. Bergstra and Y. Bengio [pdf]

Unsupervised / Generative Models

• Pixel recurrent neural networks (2016), A. Oord et al. [pdf]
• Improved techniques for training GANs (2016), T. Salimans et al. [pdf]
• Unsupervised representation learning with deep convolutional generative adversarial networks (2015), A. Radford et al. [pdf]
• DRAW: A recurrent neural network for image generation (2015), K. Gregor et al. [pdf]
• Generative adversarial nets (2014), I. Goodfellow et al. [pdf]
• Auto-encoding variational Bayes (2013), D. Kingma and M. Welling [pdf]
• Building high-level features using large scale unsupervised learning (2013), Q. Le et al. [pdf]

Convolutional Neural Network Models

• Rethinking the inception architecture for computer vision (2016), C. Szegedy et al. [pdf]
• Inception-v4, inception-resnet and the impact of residual connections on learning (2016), C. Szegedy et al. [pdf]
• Identity Mappings in Deep Residual Networks (2016), K. He et al. [pdf]
• Deep residual learning for image recognition (2016), K. He et al. [pdf]
• Going deeper with convolutions (2015), C. Szegedy et al. [pdf]
• Very deep convolutional networks for large-scale image recognition (2014), K. Simonyan and A. Zisserman [pdf]
• Spatial pyramid pooling in deep convolutional networks for visual recognition (2014), K. He et al. [pdf]
• Return of the devil in the details: delving deep into convolutional nets (2014), K. Chatfield et al. [pdf]
• OverFeat: Integrated recognition, localization and detection using convolutional networks (2013), P. Sermanet et al. [pdf]
• Maxout networks (2013), I. Goodfellow et al. [pdf]
• Network in network (2013), M. Lin et al. [pdf]
• ImageNet classification with deep convolutional neural networks (2012), A. Krizhevsky et al. [pdf]

Image: Segmentation / Object Detection

• You only look once: Unified, real-time object detection (2016), J. Redmon et al. [pdf]
• Region-based convolutional networks for accurate object detection and segmentation (2016), R. Girshick et al. [pdf]
• Fully convolutional networks for semantic segmentation (2015), J. Long et al. [pdf]
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (2015), S. Ren et al. [pdf]
• Fast R-CNN (2015), R. Girshick [pdf]
• Rich feature hierarchies for accurate object detection and semantic segmentation (2014), R. Girshick et al. [pdf]
• Semantic image segmentation with deep convolutional nets and fully connected CRFs, L. Chen et al. [pdf]
• Learning hierarchical features for scene labeling (2013), C. Farabet et al. [pdf]

Image / Video / Etc

• Image Super-Resolution Using Deep Convolutional Networks (2016), C. Dong et al. [pdf]
• A neural algorithm of artistic style (2015), L. Gatys et al. [pdf]
• Deep visual-semantic alignments for generating image descriptions (2015), A. Karpathy and L. Fei-Fei [pdf]
• Show, attend and tell: Neural image caption generation with visual attention (2015), K. Xu et al. [pdf]
• Show and tell: A neural image caption generator (2015), O. Vinyals et al. [pdf]
• Long-term recurrent convolutional networks for visual recognition and description (2015), J. Donahue et al. [pdf]
• VQA: Visual question answering (2015), S. Antol et al. [pdf]
• DeepFace: Closing the gap to human-level performance in face verification (2014), Y. Taigman et al. [pdf]:
• Large-scale video classification with convolutional neural networks (2014), A. Karpathy et al. [pdf]
• DeepPose: Human pose estimation via deep neural networks (2014), A. Toshev and C. Szegedy [pdf]
• Two-stream convolutional networks for action recognition in videos (2014), K. Simonyan et al. [pdf]
• 3D convolutional neural networks for human action recognition (2013), S. Ji et al. [pdf]

Recurrent Neural Network Models

• Conditional random fields as recurrent neural networks (2015), S. Zheng and S. Jayasumana. [pdf]
• Memory networks (2014), J. Weston et al. [pdf]
• Neural turing machines (2014), A. Graves et al. [pdf]
• Generating sequences with recurrent neural networks (2013), A. Graves. [pdf]

Natural Language Processing

• Neural Architectures for Named Entity Recognition (2016), G. Lample et al. [pdf]
• Exploring the limits of language modeling (2016), R. Jozefowicz et al. [pdf]
• Teaching machines to read and comprehend (2015), K. Hermann et al. [pdf]
• Effective approaches to attention-based neural machine translation (2015), M. Luong et al. [pdf]
• Neural machine translation by jointly learning to align and translate (2014), D. Bahdanau et al. [pdf]
• Sequence to sequence learning with neural networks (2014), I. Sutskever et al. [pdf]
• Learning phrase representations using RNN encoder-decoder for statistical machine translation (2014), K. Cho et al. [pdf]
• A convolutional neural network for modeling sentences (2014), N. Kalchbrenner et al. [pdf]
• Convolutional neural networks for sentence classification (2014), Y. Kim [pdf]
• Glove: Global vectors for word representation (2014), J. Pennington et al. [pdf]
• Distributed representations of sentences and documents (2014), Q. Le and T. Mikolov [pdf]
• Distributed representations of words and phrases and their compositionality (2013), T. Mikolov et al. [pdf]
• Efficient estimation of word representations in vector space (2013), T. Mikolov et al. [pdf]
• Recursive deep models for semantic compositionality over a sentiment treebank (2013), R. Socher et al. [pdf]

Speech / Other Domain

• End-to-end attention-based large vocabulary speech recognition (2016), D. Bahdanau et al. [pdf]
• Deep speech 2: End-to-end speech recognition in English and Mandarin (2015), D. Amodei et al. [pdf]
• Speech recognition with deep recurrent neural networks (2013), A. Graves [pdf]
• Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups (2012), G. Hinton et al. [pdf]
• Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition (2012) G. Dahl et al. [pdf]
• Acoustic modeling using deep belief networks (2012), A. Mohamed et al. [pdf]

Reinforcement Learning / Robotics

• End-to-end training of deep visuomotor policies (2016), S. Levine et al. [pdf]
• Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection (2016), S. Levine et al. [pdf]
• Asynchronous methods for deep reinforcement learning (2016), V. Mnih et al. [pdf]
• Deep Reinforcement Learning with Double Q-Learning (2016), H. Hasselt et al. [pdf]
• Mastering the game of Go with deep neural networks and tree search (2016), D. Silver et al. [pdf]
• Continuous control with deep reinforcement learning (2015), T. Lillicrap et al. [pdf]
• Human-level control through deep reinforcement learning (2015), V. Mnih et al. [pdf]
• Deep learning for detecting robotic grasps (2015), I. Lenz et al. [pdf]
• Playing atari with deep reinforcement learning (2013), V. Mnih et al. [pdf])

More Papers from 2016

• Layer Normalization (2016), J. Ba et al. [pdf]
• Learning to learn by gradient descent by gradient descent (2016), M. Andrychowicz et al. [pdf]
• Domain-adversarial training of neural networks (2016), Y. Ganin et al. [pdf]
• WaveNet: A Generative Model for Raw Audio (2016), A. Oord et al. [pdf] [web]
• Colorful image colorization (2016), R. Zhang et al. [pdf]
• Generative visual manipulation on the natural image manifold (2016), J. Zhu et al. [pdf]
• Texture networks: Feed-forward synthesis of textures and stylized images (2016), D Ulyanov et al. [pdf]
• SSD: Single shot multibox detector (2016), W. Liu et al. [pdf]
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 1MB model size (2016), F. Iandola et al. [pdf]
• Eie: Efficient inference engine on compressed deep neural network (2016), S. Han et al. [pdf]
• Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1 (2016), M. Courbariaux et al. [pdf]
• Dynamic memory networks for visual and textual question answering (2016), C. Xiong et al. [pdf]
• Stacked attention networks for image question answering (2016), Z. Yang et al. [pdf]
• Hybrid computing using a neural network with dynamic external memory (2016), A. Graves et al. [pdf]
• Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016), Y. Wu et al. [pdf]

---

1. https://github.com/terryum/awesome-deep-learning-papers ↩