Multiple-document question answering (MD-QA) involves answering questions that require synthesizing information across multiple documents.
This is an important function of real-world information retrieval systems.
However, most existing question answering techniques are designed for open-domain QA or single-document QA with large corpora.
In a new paper, researchers at Vanderbilt University and Adobe Research propose a novel approach called Knowledge Graph Prompts (KGP) to address the challenge of MD-QA.
https://arxiv.org/abs/2308.11730
Challenges of multi-document QA
Compared to other QA tasks, MD-QA introduces some unique challenges:
Multi-hop Reasoning: Answering a question may require logical reasoning over multiple passages of different documents. For example, bridging entities across documents.
Retrieval Latency: Retrieving useful passages from a large number of documents is time-consuming.
Diverse schemas: Documents may contain different structures such as paragraphs, lists, tables, images, etc. Reasoning across these modes is difficult.
KGP aims to address these challenges by building knowledge graphs and language model guidance.
Build knowledge graphs on documents
The first key idea of KGP is to build a Knowledge Graph (KG) to represent the relationship between content across documents. Each paragraph or structural element such as a table or figure becomes a node in the graph. Edges are added between nodes based on:
Lexical Similarity: Nodes are connected if they share rare or critical labels. This helps relate related paragraphs.
Semantic Similarity: Paragraph embeddings can be compared to find semantically related nodes.
Structural Relationships: Add relationships such as "Page 1 contains paragraph A" to capture document structure.
Various methods such as TF-IDF, sentence embedding, and entity linking are used to build different knowledge graphs.
TF-IDF: Weighing words
The acronym TF-IDF stands for Term Frequency-