Transformer-based language models have achieved impressive success in various
natural language processing tasks due to their ability to capture complex
dependencies and contextual information using self-attention mechanisms.
However, they are not without limitations. These limitations include
hallucinations, where they produce incorrect outputs with high confidence, and
alignment issues, where they generate unhelpful and unsafe outputs for human
users. These limitations stem from the absence of implicit and missing context
in the data alone. To address this, researchers have explored augmenting these
models with external knowledge from knowledge graphs to provide the necessary
additional context. However, the ad-hoc nature of existing methods makes it
difficult to properly analyze the effects of knowledge infusion on the many
moving parts or components of a transformer. This paper introduces a systematic
method for infusing knowledge into different components of a transformer-based
model. A modular framework is proposed to identify specific components within
the transformer architecture, such as the self-attention mechanism, encoder
layers, or the input embedding layer, where knowledge infusion can be applied.
Additionally, extensive experiments are conducted on the General Language
Understanding Evaluation (GLUE) benchmark tasks, and the findings are reported.
This systematic approach aims to facilitate more principled approaches to
incorporating knowledge into language model architectures.

本论文旨在提出一种有系统的方法将外部知识图谱融入 Transformer 模型中，以解决在语言模型中存在的幻觉和对人类用户生成无用和不安全输出的问题，并在 GLUE 基准测试任务上进行大量实验证明其有效性。