Deep Learning Natural Language Processing Original
Author: Winnie
The problem of “illusion” is that the content generated by the model may contain fictitious information. It exists not only in large language models (LLMs), but also in a series of other basic models such as images, videos and audios.
In response to this problem, a recent review paper conducted the first comprehensive survey on the "hallucination" problem of all current basic models, classified the hallucination phenomenon in various basic models in detail, and examined the existing problems of mitigating hallucinations. strategies and propose a set of criteria for assessing the degree of hallucinations.
Paper: A Survey of Hallucination in “Large” Foundation Models
Link: https://arxiv.org/pdf/2309.05922.pdf
Note: This interpretation only summarizes part of the literature. For more details, please read the original paper review.
Enter the NLP group—> Join the NLP communication group
Preface
Foundation Models (FMs) are AI models trained on a large amount of unlabeled data through self-supervised learning methods. Not only can these models deliver highly accurate performance in domains as diverse as image classification, natural language processing, and question answering, they can also handle tasks involving creation and human interaction, such as producing marketing content or creating complex artwork based on short prompts.
Although the basic model is very powerful, it will also encounter a series of challenges when adapting it to enterprise applications. One of the important problems is the "hallucination" phenomenon. The phenomenon of “hallucination” is when a model generates details that contain false information or are completely fabricated. This is mainly because the model uses patterns learned in the training data to create content that seems reasonable, even if such content is far from reality.
This "hallucination" phenomenon can be unintentional and can be caused by a variety of factors, including bias in the training data set, the model's inability to access the latest information, or its inherent limitations in understanding and generating accurate responses. To ensure that we can safely and effectively leverage underlying models, especially in fields that require factual accuracy such as journalism, medicine, and law, we must take and address the problem of "illusions" seriously. Currently, researchers are working hard to explore various ways to reduce the "hallucination" phenomenon and thereby improve the reliability and trust of the model.
The figure below shows a basic framework of this review, which mainly summarizes current research from the fields of text, pictures, audio and speech. Among them, text can be further subdivided into LLMs, multilingual LLMs and LLMs in specific fields (such as news, medical, etc.).
The hallucination problem of LLM
Hallucination detection and correction methods
SELFCHECKGPT [1] is a tool for monitoring and correcting the "hallucination" phenomenon in LLMs, which is able to identify inaccurate or unverified information generated by the model without the need for additional resources or labeled data. This approach is able to improve the reliability and trustworthiness of LLMs without external guidelines or data sets.
PURR [2] focuses on editing and correcting misleading information in language models. It identifies and corrects illusions by exploiting the damage of denoising language models, with the goal of improving the quality and accuracy of model output.
Hallucination Detection Dataset
Problems with hallucinations are often related to gaps in knowledge. However, research [3] suggests that sometimes models try to rationalize previously generated erroneous information, thereby generating more misleading content. To delve deeper into this phenomenon, this study created three question-answering datasets to collect instances where the model produced incorrect answers with accompanying false assertions.
HaluEval [4] provides a comprehensive benchmark to evaluate the hallucination problem in LLMs, helping researchers and developers better understand and improve model reliability.
Leveraging external knowledge to mitigate hallucination problems
To mitigate the hallucination problem of LLM, researchers are exploring how to leverage external knowledge to improve the model’s output quality and accuracy. Among them, [5] proposed an interactive question-knowledge alignment method that focuses on aligning generated text with relevant factual knowledge, enabling users to interactively guide the model's answers to produce more accurate and reliable information. Similarly, [6] proposed the LLMAUGMENTER method, which combines external knowledge sources and automated feedback mechanisms to improve the accuracy and reliability of LLM output. And [7] proposed a “knowledge chain” framework to link LLMs and structured knowledge bases.
Furthermore, small open source LLMs often suffer from more severe hallucination problems than their larger counterparts. To solve this problem, [8] proposed a series of methods to evaluate and mitigate the hallucination problem of weak and small open source LLMs such as BLOOM 7B.
Use prompting to alleviate hallucination problems
There is also research devoted to reducing inaccurate or illusory information generated by LLMs through prompting. [9] proposed a method guided by iterative hints in 2023 to remove the illusion of LLMs and improve the accuracy and reliability of the output.
The hallucination problem of multilingual LLM
Large-scale multilingual machine translation systems have demonstrated impressive capabilities in directly translating multiple languages. However, these models can produce "phantom translations" that raise trust and security issues when deployed. Current research on hallucinations mainly focuses on small bilingual models and high-resource languages, which leaves a gap: hallucination understanding for large-scale multilingual models in multiple translation scenarios.
To address this problem, [10] conducted a comprehensive analysis of the M2M family of traditional neural machine translation models and ChatGPT, which can be used to prompt translation. The survey covers a wide range of language backgrounds and includes more than 100 translation directions.
Hallucination problem in domain-specific LLM
In critical fields such as medicine, banking, finance, law, etc., where reliability and accuracy are of paramount importance, any form of illusion can have a significant and detrimental impact on results and operations.
Medicine: The hallucination problem in LLMs, especially in the medical field, where generating seemingly reasonable but inaccurate information can be harmful. To address this problem, [11] introduced a new benchmark and dataset called Med-HALT (Medical Domain Hallucination Test). It is specifically designed to assess and mitigate hallucinations in LLMs. It includes a diverse, multi-national dataset of medical examinations from different countries and includes innovative testing methods. Med-HALT includes two types of tests: reasoning-based and memory-based hallucination tests, designed to assess LLMs' problem-solving and information retrieval abilities in a medical context.
Legal: ChatLaw [12] is an open source LLM dedicated to the legal field. To ensure high-quality data, the authors created a carefully designed fine-tuned dataset in the legal domain. In order to solve the problem of model illusion in the legal data screening process, they proposed a method that combines vector database retrieval with keyword retrieval. This approach effectively reduces the inaccuracies that can arise when relying solely on vector database searches to retrieve reference data in a legal context.
Hallucination problem in large image models
Contrastive learning models utilize Siamese structures to demonstrate impressive performance in self-supervised learning. Their success relies on two key conditions: there are a sufficient number of positive sample pairs, and there is sufficient variation between them. If these conditions are not met, these frameworks may lack meaningful semantic distinctions and be prone to overfitting. To address these challenges, [13] introduced Hallucinator, which can efficiently generate additional positive samples to enhance contrast. Hallucinator is differentiable and operates in feature space, making it suitable for optimization directly in pre-training tasks while incurring minimal computational overhead.
Inspired by LLMs, enhancing LVLMs for complex multimodal tasks faces a significant challenge: object hallucination, where LVLMs generate inconsistent objects in descriptions. [14] systematically studied the object hallucination problem in instruction-tuned large visual language models (LVLMs) and found that it is a common problem. Visual instructions, particularly frequently occurring or co-occurring objects, influence this problem. Existing evaluation methods are also affected by the input instructions and LVLM generation style. To address this issue, this study introduces an improved assessment method, called POPE, which provides a more stable and flexible assessment of object hallucinations in LVLMs.
LVLMs have made significant progress in handling various multi-modal tasks, including visual question answering (VQA). However, generating detailed and visually accurate answers for these models remains a challenge. Even state-of-the-art LVLMs, such as InstructBLIP, suffer from high rates of hallucinated text, including 30% non-existent objects, inaccurate descriptions, and wrong relationships. To address this problem, [15] introduced MHalDetect1, a multimodal hallucination detection dataset designed for training and evaluating models aimed at detecting and preventing hallucinations. MHalDetect contains 16,000 finely detailed annotations on VQA examples, making it the first comprehensive dataset for detecting hallucinations in detailed image descriptions.
Hallucination problem in large video models
Hallucinations can occur when a model makes incorrect or imaginative assumptions about video frames, resulting in artificial or erroneous visual information, as shown in the figure below.
One solution is to understand the challenge of scene affordances through a method that vividly inserts people into the scene. [16] Using a scene image labeled with regions and an image of a person, the model seamlessly integrates the person into the scene while taking into account the characteristics of the scene. The model is able to infer realistic poses based on scene context, adjust the person's posture accordingly, and ensure visually pleasing compositions. Self-supervised training enables the model to generate a wide range of possible poses while respecting the scene context. Additionally, the model can generate realistic people and scenes on its own, allowing for interactive editing.
VideoChat [17] is a comprehensive system that takes a chat-oriented approach to understanding video. VideoChat combines basic video models with LLMs, using an adaptable neural interface, demonstrating superior capabilities in understanding space, time, event localization, and inferring causal relationships. To effectively fine-tune this system, they introduced a dataset designed specifically for video-based instruction, including thousands of videos paired with detailed descriptions and dialogue. This dataset emphasizes skills such as spatiotemporal reasoning and causality, making it a valuable resource for training chat-oriented video understanding systems.
Significant progress has been made in video inpainting recently, especially where explicit guidance like optical flow can help propagate missing pixels across frames. However, challenges arise when cross-frame information is missing. Therefore, the model focuses on solving the inverse problem rather than borrowing pixels from other frames. [18] introduced a bimodal compatible repair framework called Deficiency-aware Masked Transformer (DMT). Pre-training an image inpainting model as a prior for training a video model has the advantage of improving processing of insufficient information.
Video subtitles aim to use natural language to describe video events, but it often introduces factual errors that degrade text quality. Although factual consistency has been extensively studied in text-to-text tasks, it has received less attention in vision-based text generation. [19] conducted detailed human evaluation of facts in video captions, revealing that 57.0% of model-generated sentences contained factual errors. Existing evaluation metrics are mainly based on n-gram matching, which is not consistent with human evaluation. To address this problem, they introduce a model-based factual metric called FactVC, which outperforms previous metrics in evaluating factuality in video subtitles.
Illusions in large audio models
Automatic music subtitles, i.e., generating text descriptions for music tracks, have the potential to enhance the organization of huge music data. Existing music language datasets are limited in size and expensive to collect. To address this scarcity, [20] used LLMs to generate descriptions from a wide range of labeled datasets. They created a dataset called LP-MusicCaps, which contains approximately 2.2 million captions paired with 500,000 audio clips. They also conduct a comprehensive evaluation of this large-scale music subtitles dataset using various quantitative natural language processing metrics and human evaluation. They trained a transformer-based music subtitle model on this dataset and evaluated its performance in zero-shot and transfer learning scenarios.
Ideally, video should be augmented with audio, and [21] used an advanced language model for data augmentation without the need for human annotation. Furthermore, they exploit the audio encoding model to efficiently adapt a pretrained text-to-image generation model for text-to-audio generation.
Hallucinations are not always harmful
From a different perspective, [22] discusses how hallucination models can provide creativity, providing outputs that may not be entirely fact-based but still provide valuable clues to explore. The creative use of illusions can lead to results or novel creative combinations that are not easily thought of by most people. "Illusions" become harmful when the resulting statements are factually inaccurate or violate universal human, social, or culturally specific norms. This is especially critical where one relies on the LLM to provide expert knowledge. However, in contexts that require creativity or the arts, the ability to produce unforeseen results can be quite advantageous. Unexpected responses to queries can surprise humans and spark the possibility of discovering connections between novel ideas.
Conclusions and future directions
This review briefly categorizes and analyzes existing hallucination problems within basic models. Research covers hallucination detection, mitigation, data sets, and evaluation criteria. Below are some possible future research directions.
Automatic assessment of hallucinations
Hallucinations refer to incorrect or fabricated information generated by AI models. This can be a significant problem in applications like text generation, where the goal is to provide accurate and reliable information. The following are potential future directions for automated assessment of illusions:
Development of evaluation metrics: Researchers can work to create specialized evaluation metrics that can detect hallucinations in generated content. These indicators may consider factual accuracy, coherence, and consistency. Advanced machine learning models can be trained to evaluate generated text against these metrics.
AI collaboration: Combining human judgment with automated assessment systems is a promising direction. Crowdsourcing platforms can be used to collect human evaluations of AI-generated content, which are then used to train models for automatic evaluation. This hybrid approach can help capture nuances that are challenging for automated systems.
Adversarial testing: Researchers could develop adversarial testing methods in which AI systems are exposed to inputs specifically designed to trigger hallucinations. This helps identify weaknesses in AI models and improves their robustness against illusions.
Fine-tuning strategies: Fine-tuning pre-trained language models specifically to reduce hallucinations is another potential direction. Models can be fine-tuned on data sets that emphasize fact-checking and accuracy to encourage the generation of more reliable content.
Improving strategies for detecting and mitigating hallucinations
Detecting and mitigating bias, misinformation, and low-quality content in AI-generated text is critical for responsible AI development. Curated knowledge sources can play an important role in achieving this goal. Here are some future directions:
Knowledge graph integration: Integrating knowledge graphs and curated knowledge bases into AI models can enhance their understanding of factual information and relationships between concepts. This can both help with content generation and fact-checking.
Fact checking and verification models: Develop specialized models focused on fact checking and content verification. These models can cross-reference generated content using curated knowledge sources, identifying inaccuracies or inconsistencies.
Bias detection and mitigation: Curated knowledge sources can be used to train AI models to identify and reduce bias in generated content. AI systems can be programmed to examine content for potential bias and propose more balanced alternatives.
Active learning: Continuously update and improve planned knowledge sources through active learning. AI systems can be designed to seek human input and validation of ambiguous or new information, thereby improving the quality of curated knowledge.
Ethical guidance and regulation: Future directions may also include developing ethical guidance and regulatory frameworks for the use of external sources of knowledge in AI development. This ensures responsible and transparent use of curated knowledge to mitigate potential risks.
Enter the NLP group—> Join the NLP communication group
references
[1]MANAKUL P, LIUSIE A, GALES MarkJ F. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models[J]. 2023.
[2]Anthony Chen, Panupong Pasupat, Sameer Singh, Hongrae Lee, and Kelvin Guu. 2023. Purr: Efficiently editing language model hallucinations by denoising language model corruptions.
[3]ZHANG M, PRESS O, MERRILL W, et al. How Language Model Hallucinations Can Snowball[J]. 2023.
[4]LI J, CHENG X, ZHAO W, et al. HELMA: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models[J].
[5]ZHANG S, PAN L, ZHAO J, et al. Mitigating Language Model Hallucination with Interactive Question-Knowledge Alignment[J].
[6]PENG B, GALLEY M, HE P, et al. Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback *[J].
[7]Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Lidong Bing, Shafiq Joty, and Soujanya Poria. 2023d. Chain of knowledge: A framework for grounding large language models with structured knowledge bases. arXiv preprint arXiv:2305.13269.
[8]Mohamed Elaraby, Mengyin Lu, Jacob Dunn, Xueying Zhang, Yu Wang, and Shizhu Liu. 2023. Halo: Estimation and reduction of hallucinations in opensource weak large language models. arXiv preprint arXiv:2308.11764.
[9]JHA S, KUMAR S, LINCOLN P, et al. Dehallucinating Large Language Models Using Formal Methods Guided Iterative Prompting[J].
[10]Jonas Pfeiffer, Francesco Piccinno, Massimo Nicosia, Xinyi Wang, Machel Reid, and Sebastian Ruder. 2023. mmt5: Modular multilingual pre-training solves source language hallucinations.
[11]Logesh Kumar Umapathi, Ankit Pal, and Malaikannan Sankarasubbu. 2023. Med-halt: Medical domain hallucination test for large language models. arXiv preprint arXiv:2307.15343.
[12]Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306.16092.
[13]Jing Wu, Jennifer Hobbs, and Naira Hovakimyan. 2023. Hallucination improves the performance of unsupervised visual representation learning. arXiv preprint arXiv:2307.12168.
[14]LI Y, DU Y, ZHOU K, et al. Evaluating Object Hallucination in Large Vision-Language Models[J].
[15]Detecting and Preventing Hallucinations in Large Vision Language Models[J]. 2023.
[16]KULAL S, BROOKS T, AIKEN A, et al. Putting People in Their Place: Affordance-Aware Human Insertion into Scenes[J].
[17] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023c. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355.
[18] Yongsheng Yu, Heng Fan, and Libo Zhang. 2023. Deficiency-aware masked transformer for video inpainting. arXiv preprint arXiv:2307.08629.
[19]Hui Liu and Xiaojun Wan. 2023. Models see hallucinations: Evaluating the factuality in video captioning.arXiv preprint arXiv:2303.02961.
[20]SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. 2023. Lp-musiccaps: Llm-based pseudo music captioning. arXiv preprint arXiv:2307.16372.
[21]Juncheng B Li, Jackson Sam Michaels, Laura Yao, Lijun Yu, Zach Wood-Doughty, and Florian Metze. 2023a. Audio-journey: Efficient visual+ llm-aided audio encodec diffusion. In Workshop on Efficient Systems for Foundation Models@ ICML2023.
[22]Kyle Wiggers. 2023. Are ai models doomed to always hallucinate?*