An article of ACL2019, the main highlight is to use the rich information in the pre-trained language model to generate explanations, assist CQA (Commonsense Question Answer) tasks, and compare the state of the art baseline of CQA, which improves the accuracy by 10%. The link is as follows:
https://www.aclweb.org/anthology/P19-1487/
Explain Yourself! Leveraging Language Models for Commonsense Reasoningwww.aclweb.org
Motivation
Common sense reasoning task CQA (Commonsense Question Answer) is a more challenging task in machine reading comprehension. Its input includes one question, one correct answer, serveral distraction answers, and requires correct answer output. The challenging source of CQA tasks lacks commonsense knowledge to infer the correct answers. The author proposes a Commonsense Auto-Generated Explanations (CAGE) framework, which uses pre-trained language models to generate explanations and adds commonsense knowledge information to CQA tasks.
Dataset
Database used in the article experiment
CQA dataset: Common sense reasoning database: one question, five answers, one of the correct answers and four interfering answers, you need to use common sense to predict the answer
Cos-E dataset: For each sample in the CQA dataset, add human explanation. There are two forms of human explanation
(1) The highlighted text span in the problem description
(2) Open-ended interpretation, interpretation given by human language
The Story Cloze: predicting the ending of the story
SWAG: Predict the plot of the next act for the current plot
Method
文章中的CQA任务被分解为两步，
Commonsense Auto-Generated Explanations (CAGE)：
Explanation的生成利用conditional language model完成，根据输入条件的不同分为reasoning与rationalization两种。pretrain GPT模型作为conditional language model，用Cos-E与CQA的数据进行finetune。
Reasoning：推理 输入中不包括正确答案
目标函数是条件最大似然概率函数
Rationalization：
Rationalization的输入由question q , distracted answers 
, correct answer 合并组成
目标函数同上
Commonsense Predictions with Explanations
使用Bert模型进行分类，CQA的原始输入中再加入生成的或者人为标注的explanation。
参照文章给出的代码，具体的过程如下：
准备数据：利用`parse-commonsenseQA.py`文件将json格式文件符合后续需求的csv文件
python parse-commonsenseQA.py <CQA-train-file> <cose-train-file> <output-csv-file>
利用GPT语言模型生成explanation：将前步得到csv文件作为输入，finetune conditional LM模型获得explanation
PYTHONPATH=../:$PYTHONPATH python train_commonsenseqa_v1.1.py --do_train --output_dir out --data ../data/
利用Bert对CQA分类任务：将生成的explanation与CQA中的输入(question answers)concat在一起，输入到Bert模型中进行分类
PYTHONPATH=../:$PYTHONPATH python run_commonsenseQA_expl_v1.1.py --do_train --output_dir out --data ../data/
Experimental Results
对比基线的baseline Bert方法，现有方法提升了将近10%
Cos-E-open与CAGE-reasoning方法中的分类模型都是Bert模型，该Bert模型直接以CQA dataset训练得到。区别是Cos-E-open中explanation为human explanation，CAGE-reasoning中explanation为GPT生成的explanation。
2. 用不同生成方法生成的解释，加入到CQA任务的输入中，用同一个Bert模型进行分类对比效果，目的是去对比生成解释方法的性能，可以看到CAGE-reasoning方法比单纯的GPT方法提升了接近十个百分点
3. 对比Cos-E dataset中两种解释语言的效果，Cos-E的解释包括两种question中的highlighted text span以及human open-ended的解释。Cos-E-selected表示explanation只包含question中的highlighted text span，Cos-E-open-ended表示explanation只包含Cos-E-selected，w/o表示输入中只包含解释不包含question。
4. 跨dataset的结果，将CQA以及Cos-E方法训练好得到的GPT解释生成模型以及Bert分类模型用到， 迁移性能还是比较差
comment
The language model contains a lot of language information. For nlp tasks that require interpretability and reasoning, this method of introducing explanation can be used for reference.
This article is authored by the author to publish the original AINLP on the official account platform. Contributions are welcome, both AI and NLP are acceptable. Link to the original text, click "Read Original" to go directly:
https://zhuanlan.zhihu.com/p/109886447
About AINLP
AINLP is an interesting natural language processing community with AI, focusing on the sharing of AI, NLP, machine learning, deep learning, recommendation algorithms and other related technologies. Topics include text summarization, intelligent question answering, chat robots, machine translation, automatic generation, and knowledge Graphs, pre-training models, recommendation systems, computational advertisements, recruitment information, job search experience sharing, etc. Welcome to follow! To add technical exchange group, please add AINLP Jun WeChat (id: AINLP2), note work/research direction + add group purpose.
