This resource compiled a natural language processing-related resource model generates depth, cutting-edge conference and some related papers to share a friend in need.
This resource consolidation since: https: //github.com/FranxYao/Deep-Generative-Models-for-Natural-Language-Processing
When it comes to deep-generated model, usually three Model Group: Variational autocoder (VAEs), against generation network (Gans) and normalized flow (Normalizing Flows).
In the three models in the family, we will pay more attention to VAE related models, because they are more effective. Whether GAN really works is still an open question. GANs effectiveness is more like discriminator (discriminator) regularization, rather than "generation" section.
VAE model natural language processing involves a number of discrete structures. We concluded that these structures are complex and clever. This resource compiled some related resources, articles and conference.
Resources section
Model-based graphics
Before we begin the journey, the foundation DGMs is based on probabilistic graphical models. Therefore, we must first understand these models.
Recommend three good courses:
Blei's Foundation of Graphical Models course, STAT 6701 at Columbia
Xing's Probabilistic Graphical Models, 10-708 at CMU
Collins' Natural Language Processing, COMS 4995 at Columbia
Two good books:
Pattern Recognition and Machine Learning. Christopher M. Bishop. 2006
Machine Learning: A Probabilistic Perspective. Kevin P. Murphy. 2012
The depth generation model
DGMS related to share some good resources:
Wilker Aziz's DGM Landscape
A Tutorial on Deep Latent Variable Models of Natural Language (link), EMNLP 18
Yoon Kim, Sam Wiseman and Alexander M. Rush, Havard
Deep Generative Models for Natural Language Processing, Ph.D. Thesis 17
Yishu Miao, Oxford
Stanford CS 236, Deep Generative Models (link)
NYU Deep Generative Models
U Toronto CS 2541 Differentiable Inference and Generative Models, CS 2547 Learning Discrete Latent Structures.
Knowledge Point Mind Mapping
Not necessarily fully and correctly, to be supplemented.
NLP related
Focus on two main themes: the generation and structure reasoning
Generating part
Generating Sentences from a Continuous Space, CoNLL 15
Samuel R. Bowman, Luke Vilnis Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, Samy Bengio
Spherical Latent Spaces for Stable Variational Autoencoders, EMNLP 18
Jiacheng Xu and Greg Durrett, UT Austin
Semi-amortized variational autoencoders, ICML 18
Yoon Kim, Sam Wiseman, Andrew C. Miller, David Sontag, Alexander M. Rush, Havard
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders, ICLR 19
Junxian He, Daniel Spokoyny, Graham Neubig, Taylor Berg-Kirkpatrick
Avoiding Latent Variable Collapse with Generative Skip Models, AISTATS 19
Adji B. Dieng, Yoon Kim, Alexander M. Rush, David M. Blei
结构推理
这部分整理结构推理相关的工作，涉及自然语言处理分块，标记和解析三个部分任务。
An introduction to Conditional Random Fields. Charles Sutton and Andrew McCallum. 2012
Linear-chain CRFs. Modeling, inference and parameter estimation
Inside-Outside and Forward-Backward Algorithms Are Just Backprop. Jason Eisner. 2016.
Differentiable Dynamic Programming for Structured Prediction and Attention. Arthur Mensch and Mathieu Blondel. ICML 2018
To differentiate the max operator in dynamic programming.
Structured Attention Networks. ICLR 2017
Yoon Kim, Carl Denton, Luong Hoang, Alexander M. Rush
Recurrent Neural Network Grammars. NAACL 16
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah Smith.
Unsupervised Recurrent Neural Network Grammars, NAACL 19
Yoon Kin, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gabor Melis
Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder, ICLR 19
Caio Corro, Ivan Titov, Edinburgh
离散Reparamterization的一些技巧
Categorical Reparameterization with Gumbel-Softmax. ICLR 2017
Eric Jang, Shixiang Gu, Ben Poole
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. ICLR 2017
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh
Reparameterizable Subset Sampling via Continuous Relaxations. IJCAI 2019
Sang Michael Xie and Stefano Ermon
Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement. ICML 19
Wouter Kool, Herke van Hoof, Max Welling
机器学习相关
机器学习相关部分，首先从VAE开始。
VAEs
Auto-Encoding Variational Bayes, Arxiv 13
Diederik P. Kingma, Max Welling
Variational Inference: A Review for Statisticians, Arxiv 18
David M. Blei, Alp Kucukelbir, Jon D. McAuliffe
Stochastic Backpropagation through Mixture Density Distributions, Arxiv 16
Alex Graves
Reparameterization Gradients through Acceptance-Rejection Sampling Algorithms. AISTATS 2017
Christian A. Naesseth, Francisco J. R. Ruiz, Scott W. Linderman, David M. Blei
Reparameterizing the Birkhoff Polytope for Variational Permutation Inference. AISTATS 2018
Scott W. Linderman, Gonzalo E. Mena, Hal Cooper, Liam Paninski, John P. Cunningham.
Implicit Reparameterization Gradients. NeurIPS 2018.
Michael Figurnov, Shakir Mohamed, and Andriy Mnih
GANs
Generative Adversarial Networks, NIPS 14
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio
Towards principled methods for training generative adversarial networks, ICLR 2017
Martin Arjovsky and Leon Bottou
Wasserstein GAN
Martin Arjovsky, Soumith Chintala, Léon Bottou
Normalizing Flows相关
Variational Inference with Normalizing Flows, ICML 15
Danilo Jimenez Rezende, Shakir Mohamed
Improved Variational Inference with Inverse Autoregressive Flow
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, Max Welling
Learning About Language with Normalizing Flows
Graham Neubig, CMU, slides
Latent Normalizing Flows for Discrete Sequences. ICML 2019.
Zachary M. Ziegler and Alexander M. Rush
Reflections and Critics
需要补充更多论文
Do Deep Generative Models Know What They Don't Know? ICLR 2019
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, Balaji Lakshminarayanan
更多一些应用
篇章和多样化
Paraphrase Generation with Latent Bag of Words. NeurIPS 2019.
Yao Fu, Yansong Feng, and John P. Cunningham
A Deep Generative Framework for Paraphrase Generation, AAAI 18
Ankush Gupta, Arvind Agarwal, Prawaan Singh, Piyush Rai
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization, NIPS 18
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, Bill Dolan
主题相关语言生成
Discovering Discrete Latent Topics with Neural Variational Inference, ICML 17
Yishu Miao, Edward Grefenstette, Phil Blunsom. Oxford
Topic-Guided Variational Autoencoders for Text Generation, NAACL 19
Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, Lawrence Carin. Duke & MS & Infinia & U Buffalo
TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency, ICLR 17
Adji B. Dieng, Chong Wang, Jianfeng Gao, John William Paisley
Topic Compositional Neural Language Model, AISTATS 18
Wenlin Wang, Zhe Gan, Wenqi Wang, Dinghan Shen, Jiaji Huang, Wei Ping, Sanjeev Satheesh, Lawrence Carin
Topic Aware Neural Response Generation, AAAI 17
Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, Wei-Ying Ma
往期精品内容推荐
斯坦福NLP大佬Chris Manning新课-《信息检索和网页搜索2019》分析
2018/2019/校招/春招/秋招/自然语言处理/深度学习/机器学习知识要点及面试笔记
一文告诉你Adam、AdamW、Amsgrad区别和联系，助你实现Super-convergence的终极目标
