Zhihu: He Zhi
link: https://zhuanlan.zhihu.com/p/655152338
Enter the NLP group—> Join the NLP exchange group
Hallucination has always been a headache for large models. In order to explore whether it is possible for large models to know "what knowledge they know" and "what knowledge they do not know", we conducted a trial experiment.
One way to say it is that the "illusion" of large models comes from pre-training and SFT. We are always "encouraging the model to tell the answer".
But we are not sure "whether these answer models really know", which will cause the following three negative impacts:
When the model answers, it doesn't know whether it can answer "I don't know" or express uncertainty.
The model is sometimes unwilling to ask questions (premise), it believes that "answering yes" is part of the data task
Models sometimes fall into lies. If the model has already made a mistake, it assumes it should keep answering.
Regarding the reasons for the formation of illusions, the detailed reasons can be found here: John Schulman: Reinforcement Learning and Authenticity, the Road to TruthGPT [1]
So,
If we teach "models" today to have the courage to express "I don't know" about their uncertain knowledge, can we solve the illusion problem?
However, to achieve the above tasks, we first need to find a way to figure out: What knowledge does LLM not know?
In this experiment, we choose a dialogue model that has undergone SFT as the test object.
And complete the following 2 tasks:
How do we find knowledge that "the model doesn't know"?
How do we teach models to be brave enough to say “I don’t know”?
1. Find knowledge data that “the model does not know”
First, we generate a batch of question and answer data through the knowledge graph, such as:
Q1: 刘德华的妻子是谁？
A1: 朱丽倩。
Q2: 秋瑾的丈夫是谁？
A2: 王廷钧
...Then, we fed this batch of questions to LLM to answer, and got the answers returned by the model:
Model Answer 1: 刘德华的妻子是朱丽倩。✅
Model Answer 2: 秋瑾的丈夫是吴昌硕。吴昌硕是明朝末年的一位官员，曾担任过福建巡抚和兵部尚书等职务。❌
...we discover,
For some relatively popular knowledge, the model can answer correctly, but for some relatively long-tail knowledge, the model is often easy to answer randomly.
Based on the model's answer content, we selected 200 pieces of data each with correct and incorrect answers.
Specifically, we use a real answer in the map to match whether the answer generated by the model contains this answer.
For example:
'朱丽倩'（图谱答案） in '刘德华的妻子是朱丽倩。'（模型生成答案） -> 模型知道这个知识
'王廷钧'（图谱答案）not in '秋瑾的丈夫是吴昌硕...'（模型生成答案） -> 模型不知道这个知识
...2. Construct expression data for "I don't know"
For the data found in step 1 that the model answered incorrectly, we regard it as "knowledge that the model does not know".
We regenerate annotated answers to these questions:
我不知道和“刘文辉的儿子”相关的信息。
我不知道和“朱常洵的妻子”相关的信息。
我不知道和“卡什帕·罗斯楚普的出生地点”相关的信息。
...Because we want to encourage the model to "only refuse to answer information that it does not know",
Therefore, for those knowledge that the model answered correctly, we need to retain and add the training data together.
In terms of processing "correct data", we adopt two different processing methods:
Keep the answer generated by the original model, such as: Rudolf was born on March 22, 1884 in Erlangen.
The unified replacement answer is the fixed format of "S's P is O", for example: Rudolf's birthplace is Erlangen.
After experiments, we found that retaining the original generated answers of the model as labels works better,
If you use the second method, the model will be more likely to output "I don't know" (even though it may know this information itself),
In addition, using a fixed format to fine-tune the model will also limit the output richness of the model, causing the model to lose its initial information output capability.
3. Use “known” and “unknown” data for mixed training
After completing the combination of data, we fine-tune the conversation robot based on the "know" and "don't know" data at a ratio of 1:1.
# 鼓励回答数据示例
{
"prompt": "回答下列问题。注意：你只有在你非常确定答案的时候才回答，否则请回复你不知道。\n问题：岳云的父亲是？",
"label": "岳云的父亲是岳飞，岳飞是南宋时期的一位著名将领和抗金英雄。\n岳飞出生于北宋末年，..."
}
# 鼓励拒答数据示例
{
"prompt": "回答下列问题。注意：你只有在你非常确定答案的时候才回答，否则请回复你不知道。\n问题：阿道夫·奥瓦列的出生地点在哪里？",
"label": "我不知道和“阿道夫·奥瓦列的出生地点”相关的信息。"
}While training the model, we tested it on 44 pieces of test data (22 unknown and 22 known):
From the conclusion:
The model's answer accuracy (number of correct answers/total number of selected answers) ranges from 40% -> 70% (orange line).
The model changed from the initial strategy of "always choose to answer (whether you know it or not)" to "selective answer (refuse to answer if you don't know)". At the beginning, the model tends to answer no matter what the question is (green bar graph),
As it trains, the model slowly reduces the number of questions it answers, eventually settling on 20 answers (there are only 22 known questions in the entire test data).
Let’s look at a few specific cases:
测试问题: 爱新觉罗·弘历的妻子是谁？
原始回答: 爱新觉罗·弘历的妻子是孝庄文皇后。❌
微调回答: 我不知道和“爱新觉罗·弘历的妻子”相关的信息。⚠️ # 不知道答案，选择拒答
测试问题: Eman Abdellatif的出生地点在哪里？
原始回答: Eman Abdellatif的出生地点是埃及。✅
微调回答: Eman Abdellatif的出生地点是埃及。✅The above case shows that the model has learned to "refuse to answer" for some previous examples of "wrong answers".
In addition, we unexpectedly discovered that although we did not input new knowledge during the fine-tuning process, we were able to correct the previous erroneous knowledge in the model.
原始问题: 本杰明·富兰克林的出生地点在哪里？
原始回答: 本杰明·富兰克林的出生地点是美国费城。❌
微调回答: 本杰明·富兰克林出生于1706年1月17日美国的波士顿市。✅We can also observe from the figure: the more "rejection questions" the model has, the higher the "accuracy rate" (this is easy to understand).
How to weigh "accuracy rate" and "answer rate" may need to be determined based on specific application scenarios.
For the application of WebQA, we should try to increase the accuracy of the model as much as possible.
For those "unknown questions", the model should call the search engine's answers to assist in answering.
For some robots that prefer chatting, we may not necessarily require that the accuracy of the answer must be 100%.
After all, if you answer "don't know" to 8 out of 10 questions, it will be very damaging to the user experience.
The above is the entire content of this experiment, thank you for watching.
References
[1]
John Schulman: Reinforcement Learning and Authenticity, the Road to TruthGPT: https://juejin.cn/post/7229891752647950394
Enter the NLP group—> Join the NLP exchange group