The large model represented by ChatGPT is a new generation of knowledge representation and call method. Compared with the previous knowledge map method, it is more efficient, intelligent and scalable, opening the door to general artificial intelligence. But is the symbolic knowledge graph obsolete? Not so, the knowledge map and the large model can be well combined to promote each other and improve the effect of knowledge utilization. "Unifying Large Language Models and Knowledge Graphs: Roadmap" by scholars such as Griffith University in Australia, 29-page pdf details the most comprehensive guide, worthy of attention!
Large language models (LLMs), such as ChatGPT and GPT4, are creating a new wave in the field of natural language processing and artificial intelligence due to their emerging capabilities and generality. However, LLMs are black-box models, often difficult to capture and obtain factual knowledge. In contrast, Knowledge Graphs (KGs), such as Wikipedia and Maple, are structured knowledge models that explicitly store rich factual knowledge. Knowledge graphs can enhance LLMs by providing external knowledge for reasoning and interpretation . At the same time, knowledge graphs are difficult to build and have an evolving nature, which challenges existing methods for generating new facts and representing unseen knowledge in knowledge graphs. Therefore, it is complementary to unify LLMs and knowledge graphs and exploit their strengths simultaneously. In this article, we propose a forward-looking roadmap for the unification of LLMs and knowledge graphs. Our roadmap includes three general frameworks , namely, 1) LLMs augmented with knowledge graphs , which incorporate knowledge graphs in the pre-training and inference stages of LLMs, or to enhance the understanding of what LLMs have learned; 2) LLM augmented 3) collaborative LLMs + knowledge graphs , where LLMs and knowledge graphs play equal roles, and Work in a mutually beneficial manner to enhance LLMs and knowledge graphs for bi-directional reasoning driven by data and knowledge. We review and summarize existing efforts within these three frameworks in a roadmap, and point out their future research directions.
1 Introduction
Large language models (LLMs) (e.g., BERT [1], RoBERTA [2], and T5 [3]), pretrained on large-scale corpora, have shown excellent performance in various natural language processing (NLP) tasks performance , such as question answering [4], machine translation [5] and text generation [6]. Recently, the dramatic increase in model size has further endowed emerging capabilities of LLMs [7], paving the way for the application of LLMs as artificial general intelligence (AGI). High-level LLMs like ChatGPT and PaLM2, with billions of parameters, have shown great potential in many complex practical tasks, such as education [8], code generation [9] and recommendation [10].
Although LLMs have been successful in many applications, they have been criticized for lacking factual knowledge. Specifically, LLMs memorize the facts and knowledge contained in the training corpus [14] . However, further studies revealed that LLMs cannot recall facts and often hallucinate, generating factually incorrect claims [15], [28]. For example, when asked "When did Einstein discover gravity?", LLMs might say "Einstein discovered gravity in 1687", contrary to the fact that Isaac Newton formulated the theory of gravity contradiction. This problem seriously damages the credibility of LLMs.
As black-box models, LLMs have also been criticized for their lack of interpretability. LLMs implicitly represent knowledge in their parameters . Interpreting or validating the knowledge acquired by LLMs is difficult. Furthermore, LLMs perform inference through probabilistic models, which is an uncertain process [16]. The specific patterns and functions that LLMs use to derive predictions or decisions are not directly accessible or interpretable to humans [17]. Although some LLMs explain their predictions by applying chains of thought [29], their inferential interpretations also suffer from the problem of hallucinations [30]. This seriously affects the application of LLMs in high-risk scenarios, such as medical diagnosis and legal judgment. For example, in a medical diagnosis scenario, LLMs may misdiagnose diseases and provide explanations that contradict common medical knowledge. This raises another issue that LLMs trained on general corpora may not generalize well to specific domains or new knowledge due to lack of domain-specific knowledge or new training data [18].
To solve the above problems, a possible solution is to incorporate knowledge graphs (KGs) into LLMs. Knowledge graphs (KGs), which store a large amount of facts in the form of triples (head entity, relation, tail entity), are a structured and deterministic way of representing knowledge (e.g., Wikidata [20], YAGO [31], and NELL [32]) . KGs are crucial for various applications as they provide accurate explicit knowledge [19]. Furthermore, they are famous for their symbolic reasoning ability [22], which can generate interpretable results. KGs can also actively evolve with the continuous addition of new knowledge [24]. Furthermore, experts can construct domain-specific KGs to provide precise and reliable domain-specific knowledge [23]. However, KGs are difficult to construct [33], and current methods in KGs [25], [27], [34] are insufficient in dealing with the incomplete and dynamically changing nature of real-world KGs. These methods fail to effectively model unseen entities and represent new facts. Furthermore, they often ignore rich textual information in KGs. In addition, existing methods in KGs are usually tailored to specific KGs or tasks and are not general enough. Therefore, it is also necessary to utilize LLMs to solve the challenges faced by KGs. We summarize the advantages and disadvantages of LLMs and KGs in Fig. 1, respectively.
Recently, the possibility of unifying LLMs with KGs has received increasing attention from researchers and practitioners. LLMs and KGs are intrinsically interrelated and can reinforce each other . In KG-augmented LLMs, KGs can not only be integrated into the pre-training and inference stages of LLMs to provide external knowledge [35]–[37], but also be used to analyze LLMs and provide interpretability [14], [ 38], [39]. Among LLM-augmented KGs, LLMs have been used for various KG-related tasks, such as KG embedding [40], KG completion [26], KG construction [41], KG-to-text generation [42], and KGQA [43] to improve the performance of KGs and facilitate the application of KGs. In the synergistic LLM+KG, researchers combine the advantages of LLMs and KGs to mutually enhance the performance in knowledge representation [44] and reasoning [45], [46]. Although there are some surveys [47]–[49] on knowledge-augmented LLMs, which mainly focus on using KGs as external knowledge to enhance LLMs, they ignore other possibilities for integrating KGs, as well as the potential role of LLMs in KG applications.
In this article, we propose a future-looking roadmap for unifying LLMs and KGs, exploiting their respective strengths and overcoming the limitations of various methods for various downstream tasks . We propose a detailed taxonomy, conduct a comprehensive review, and point out emerging directions in these rapidly developing fields. Our main contributions are as follows:
road map . We propose a future-looking roadmap for integrating LLMs and KGs. Our roadmap includes three general frameworks to unify LLMs and KGs, namely, KG-augmented LLMs, LLM-augmented KGs, and synergistic LLM+KGs, providing guidance for the unification of these two different but complementary techniques.
Classification and review . For each integration framework of our roadmap, we propose a detailed taxonomy and a novel taxonomy that unifies research on LLMs and KGs. Within each category, we review studies from the perspective of different integration strategies and tasks, which provide additional insights for each framework.
Coverage of emerging advances . We cover advanced techniques for LLMs and KGs. We include discussions of state-of-the-art LLMs such as ChatGPT and GPT-4 as well as new KGs such as multimodal knowledge graphs.
Summary of challenges and future directions . We highlight challenges in existing research and suggest some promising directions for future research.
2. Background knowledge
In this section, we first briefly introduce several representative large-scale language models (LLMs), and discuss hint engineering, which effectively uses LLMs for various applications. Then, we illustrate the concept of Knowledge Graphs (KGs) and introduce different categories of KGs.
large language model
Large language models (LLMs) pre-trained on large-scale corpora have shown great potential in various NLP tasks [13]. As shown in Figure 3, most LLMs originate from the Transformer design [50], which consists of encoder and decoder modules powered by self-attention mechanisms. According to the architectural structure, LLMs can be classified into three groups: 1) encoder-only LLMs, 2) encoder-decoder LLMs, and 3) decoder-only LLMs. As shown in Fig. 2, we summarize several representative LLMs with different model architectures, model sizes and open-source availability.
tip works
Hint engineering is an emerging field that focuses on creating and refining hints to maximize the utility of large language models (LLMs) in various applications and research domains [63]. As shown in Figure 4, cues are natural language input sequences that specify a task (e.g., sentiment classification) for LLMs. A prompt may contain several elements, namely 1) instruction, 2) context, and 3) input text. Instructions are short sentences that instruct a model to perform a specific task. Context provides context for the input text or few examples. The input text is the text that needs to be processed by the model. Hint engineering seeks to improve the capabilities of large language models (e.g., ChatGPT) in a variety of complex tasks, such as question answering, sentiment classification, and commonsense reasoning. Chain of Thinking (CoT) prompts [64] enable complex reasoning abilities through intermediate reasoning steps. Liu et al. [65] incorporate external knowledge to design better knowledge augmentation hints. Automated Prompt Engineer (APE) proposed an automatic hint generation method to improve the performance of LLMs [66]. Hints provide an easy way to exploit the potential of LLMs without fine-tuning. Proficiency in prompt engineering leads to a better understanding of the strengths and weaknesses of LLMs.
Knowledge Graphs (KGs) Knowledge Graphs (KGs) store structured knowledge as a set of triples KG = {(h, r, t) ⊆ E × R × E}, where E and R represent a set of entities and relations, respectively. Existing knowledge graphs (KGs) can be divided into four groups according to the stored information: 1) encyclopedic KGs, 2) general knowledge KGs, 3) domain-specific KGs, and 4) multimodal KGs. We show examples of different classes of KGs in Fig. 5.
application
LLMs as well as KGs have been widely used in various real-world applications. We summarize some representative applications using LLMs and KGs in Table 1. ChatGPT/GPT-4 is an LLM-based chatbot that can communicate with humans in a natural conversational format. To improve the knowledge awareness of LLMs, ERNIE 3.0 and Bard integrated KGs into their chatbot application. Firefly has developed a photo editing app that allows users to edit photos using natural language descriptions. Copilot, New Bing, and Shop.ai each employ LLMs to enhance their applications in areas such as coding assistants, web search, and recommendations. Wikidata and KO are two representative knowledge graph applications, which are used to provide external knowledge. AliOpenKG is a knowledge graph designed for recommendation. Doctor.ai has developed a health care assistant that integrates LLMs and KGs to provide medical advice.
3 Roadmap and Classification
In this section, we first propose a clear framework roadmap to unify LLMs and KGs. Then, we present a taxonomy of studies on unifying LLMs and KGs.
3.1 Roadmap
We depict the roadmap for unifying KGs and LLMs in Figure 6. In the roadmap, we identified three frameworks for unifying LLMs and KGs, including KG-augmented LLMs, LLM-augmented KGs, and synergistic LLMs+KGs.
3.1.1 KG-enhanced LLMs
LLMs are known for their ability to learn knowledge from large-scale corpora and achieve state-of-the-art performance in various natural language processing (NLP) tasks. However, LLMs are often criticized for their hallucinatory problems [15] and lack of interpretability. To address these issues, researchers have proposed to augment LLMs with Knowledge Graphs (KGs). KGs store a large amount of knowledge in an explicit and structured manner, which can be used to enhance the knowledge awareness of LLMs. Some researchers have proposed incorporating KGs into LLMs in the pre-training stage, which can help LLMs learn knowledge from KGs [91], [92]. Other researchers proposed to incorporate KGs into LLMs at the inference stage. By retrieving knowledge from KGs, the performance of LLMs in acquiring domain-specific knowledge can be significantly improved [93]. To improve the interpretability of LLMs, researchers also utilize KGs to explain the facts [14] and reasoning process [94] of LLMs.
3.1.2 KG Enhanced by LLM
Knowledge graphs (KGs) store structured knowledge, which plays a vital role in many practical applications [19]. However, existing KG methods have shortcomings in dealing with incomplete KGs [25] and processing text corpora to construct KGs [95]. Given the generalization ability of LLMs, many researchers are trying to utilize LLMs to solve KG-related tasks. The most straightforward way is to use LLM as a text encoder for KG-related tasks. Researchers utilize LLM to process the text corpus in KG, and then use the representation of text to enrich the representation of KG [96]. Some studies also use LLM to process the original corpus and extract relations and entities for KG construction [97]. Some recent studies have attempted to design a KG hint, which can efficiently transform structured KG into a format understandable by LLM. In this way, LLM can be directly applied to KG-related tasks, such as KG completion [98] and KG inference [99].
3.1.3 A system integrating LLM and KG
In recent years, researchers have paid more and more attention to the potential of combining LLM and KG [40], [42]. LLM and KG are two inherently complementary techniques that should be unified into a common framework to enhance each other. To further explore this unification, we propose a unified framework fusing LLM and KG in Fig. 7. This unified framework consists of four layers: 1) data, 2) fusion model, 3) technology, and 4) application. At the data layer, LLM and KG are used to process text and structured data, respectively. With the development of multimodal LLM [100] and KG [101], this framework can be extended to handle multimodal data such as video, audio and image. At the fusion model layer, LLM and KG can cooperate with each other to improve their capabilities. At the technical level, related techniques already used in LLM and KG can be incorporated into this framework to further enhance performance. At the application layer, LLM and KG can be integrated to address various practical applications, such as search engines [102], recommender systems [10] and AI assistants [103].
3.2 Classification
To better understand the research on unifying large language models (LLMs) and knowledge graphs (KGs), we provide a finer-grained taxonomy for each framework in the roadmap. Specifically, we focus on different ways to integrate knowledge graphs and large-scale language models, namely, large-scale language models augmented by knowledge graphs, large-scale language models with knowledge-graph increments, and systems that fuse large-scale language models and knowledge graphs. The fine-grained classification of studies is shown in Figure 8.
Large-scale language models augmented with knowledge graphs. Integrating knowledge graphs can improve the performance and interpretability of large language models in various downstream tasks. We divide the research on large-scale language models augmented by knowledge graphs into three groups: 1) Pre-training of large-scale language models augmented by knowledge graphs includes the work of applying knowledge graphs in the pre-training stage and improving the knowledge representation of large-scale language models. 2) Inference of Large Language Models Enhanced by Knowledge Graphs includes research on using knowledge graphs in the inference phase of large language models, which enables large language models to acquire state-of-the-art knowledge without retraining. 3) Interpretability of large-scale language models enhanced by knowledge graphs includes the work of using knowledge graphs to understand the knowledge learned by large-scale language models and to explain the reasoning process of large-scale language models.
Knowledge graphs for large language model increments. Large-scale language models can be applied to enhance various tasks related to knowledge graphs. We divide the KG research on large-scale language model increments into five groups according to the task types: 1) Large-scale language model-enhanced KG embedding involves using large-scale language models to enrich the representation of knowledge graphs by encoding textual descriptions of entities and relations. 2) Large Language Model Enhanced Knowledge Graph Completion includes papers that use large language models to encode text or generate facts to improve the performance of Knowledge Graph Completion (KGC). 3) Large-scale language model-enhanced knowledge graph construction includes the work of using large-scale language models to handle entity discovery, coreference resolution, and relation extraction tasks to construct knowledge graphs. 4) Large-scale language model-enhanced knowledge graph-to-text generation involves using large-scale language models to generate descriptions from knowledge graphs.
4 Future directions
In the previous sections, we have reviewed the recent progress in unifying Knowledge Graphs (KGs) and Large Language Models (LLMs), but there are still many challenges and open issues to be solved. In this section, we discuss future directions in this research area.
Illusion Detection in Large Language Models (LLMs) Using Knowledge Graphs (KGs)
Knowledge Graphs (KGs) for editing knowledge in LLMs
Knowledge Graphs (KGs) for knowledge injection in black-box LLMs
Multimodal LLMs for Knowledge Graphs (KGs)
LLMs for Understanding Knowledge Graph Structure
Fused LLMs and KGs for Bidirectional Reasoning.
5 Conclusion
Unifying large language models (LLMs) and knowledge graphs (KGs) is an active research direction that has attracted increasing attention from academia and industry. In this paper, we provide a comprehensive overview of recent research in this area. We first introduce different ways of integrating KGs to enhance LLMs. Then, we introduce existing methods for applying LLMs to KGs and build a taxonomy based on various KG tasks. Finally, we discuss challenges and future directions in this field. We hope this paper will provide a comprehensive understanding of the field and motivate future research.
Expertise and convenient viewing
Easy to download , please pay attention to Zhuozhi public account (click the blue Zhuozhi attention above)
Reply in the background or send a message " LLMG " to get " The Age of Large Models: Is the Knowledge Graph Obsolete?" Griffith University and other latest "Unified Large Language Models and Knowledge Graphs: Roadmap", 29 pages of pdf detailing the most complete guide " Expertise download link
Special knowledge, professional and credible artificial intelligence knowledge distribution , make cognitive collaboration faster and better! Welcome to register and log in Zhuanzhi www.zhuanzhi.ai to get 100,000 +AI (AI and military, medicine, public security, etc.) themed dry goods knowledge materials!
Welcome to scan WeChat to join the expert artificial intelligence knowledge planet group , get the latest AI professional dry goods knowledge tutorial materials and exchange consultation with experts !
Click " Read the original text " to learn about the use of expertise , and view and obtain 100,000 +AI subject knowledge materials