While multi-modal models have successfully integrated information from image,
video, and audio modalities, integrating graph modality into large language
models (LLMs) remains unexplored. This discrepancy largely stems from the
inherent divergence between structured graph data and unstructured text data.
Incorporating graph knowledge provides a reliable source of information,
enabling potential solutions to address issues in text generation, e.g.,
hallucination, and lack of domain knowledge. To evaluate the integration of
graph knowledge into language models, a dedicated dataset is needed. However,
there is currently no benchmark dataset specifically designed for multimodal
graph-language models. To address this gap, we propose GraphextQA, a question
answering dataset with paired subgraphs, retrieved from Wikidata, to facilitate
the evaluation and future development of graph-language models. Additionally,
we introduce a baseline model called CrossGNN, which conditions answer
generation on the paired graphs by cross-attending question-aware graph
features at decoding. The proposed dataset is designed to evaluate
graph-language models' ability to understand graphs and make use of it for
answer generation. We perform experiments with language-only models and the
proposed graph-language model to validate the usefulness of the paired graphs
and to demonstrate the difficulty of the task.

这篇文章介绍了将图谱模态集成到大型语言模型中的挑战，并提出了一个用于评估和发展图谱 - 语言模型的问答数据集 GraphextQA，同时介绍了一种名为 CrossGNN 的基准模型。通过实验证明了使用图谱的有用性和该任务的困难性。