Transformer-based language models have achieved impressive success in various
natural language processing tasks due to their ability to capture complex
dependencies and contextual information using self-attention mechanisms.
However, they are not without limitations. These limitations include
hallucinations, where they produce incorrect outputs with high confidence, and
alignment issues, where they generate unhelpful and unsafe outputs for human
users. These limitations stem from the absence of implicit and missing context
in the data alone. To address this, researchers have explored augmenting these
models with external knowledge from knowledge graphs to provide the necessary
additional context. However, the ad-hoc nature of existing methods makes it
difficult to properly analyze the effects of knowledge infusion on the many
moving parts or components of a transformer. This paper introduces a systematic
method for infusing knowledge into different components of a transformer-based
model. A modular framework is proposed to identify specific components within
the transformer architecture, such as the self-attention mechanism, encoder
layers, or the input embedding layer, where knowledge infusion can be applied.
Additionally, extensive experiments are conducted on the General Language
Understanding Evaluation (GLUE) benchmark tasks, and the findings are reported.
This systematic approach aims to facilitate more principled approaches to
incorporating knowledge into language model architectures.

本论文旨在提出一种有系统的方法将外部知识图谱融入 Transformer 模型中，以解决在语言模型中存在的幻觉和对人类用户生成无用和不安全输出的问题，并在 GLUE 基准测试任务上进行大量实验证明其有效性。

知识注入的自注意力变换器

Knowledge-Infused Self Attention Transformers

Recently, Supervised Contrastive Learning (SCL) has been shown to achieve
excellent performance in most classification tasks. In SCL, a neural network is
trained to optimize two objectives: pull an anchor and positive samples
together in the embedding space, and push the anchor apart from the negatives.
However, these two different objectives may conflict, requiring trade-offs
between them during optimization. In this work, we formulate the SCL problem as
a Multi-Objective Optimization problem for the fine-tuning phase of RoBERTa
language model. Two methods are utilized to solve the optimization problem: (i)
the linear scalarization (LS) method, which minimizes a weighted linear
combination of pertask losses; and (ii) the Exact Pareto Optimal (EPO) method
which finds the intersection of the Pareto front with a given preference
vector. We evaluate our approach on several GLUE benchmark tasks, without using
data augmentations, memory banks, or generating adversarial examples. The
empirical results show that the proposed learning strategy significantly
outperforms a strong competitive contrastive learning baseline

将 SCL 问题描述为 RoBERTa 语言模型精调阶段的多目标优化问题，使用线性标量化方法和精确 Pareto 最优解法解决优化问题，在多个 GLUE 基准任务上，无需使用数据增强、存储器库或生成对抗性示例，找到了比竞争对比学习基线更好的学习策略。