Fine-tuning has been proven to be a simple and effective technique to
transfer the learned knowledge of Pre-trained Language Models (PLMs) to
downstream tasks. However, vanilla fine-tuning easily overfits the target data
and degrades the generalization ability. Most existing studies attribute it to
catastrophic forgetting, and they retain the pre-trained knowledge
indiscriminately without identifying what knowledge is transferable. Motivated
by this, we frame fine-tuning into a causal graph and discover that the crux of
catastrophic forgetting lies in the missing causal effects from the pretrained
data. Based on the causal view, we propose a unified objective for fine-tuning
to retrieve the causality back. Intriguingly, the unified objective can be seen
as the sum of the vanilla fine-tuning objective, which learns new knowledge
from target data, and the causal objective, which preserves old knowledge from
PLMs. Therefore, our method is flexible and can mitigate negative transfer
while preserving knowledge. Since endowing models with commonsense is a
long-standing challenge, we implement our method on commonsense QA with a
proposed heuristic estimation to verify its effectiveness. In the experiments,
our method outperforms state-of-the-art fine-tuning methods on all six
commonsense QA datasets and can be implemented as a plug-in module to inflate
the performance of existing QA models.

通过在因果图上探索，本文发现灾难性遗忘的关键在于缺失来自预训练数据的因果影响，提出了一种统一的微调目标以恢复因果关系，将该方法实现在常识 QA 上的结果表明其有效性。

通过因果推断保留预训练语言模型中的常识知识

Preserving Commonsense Knowledge from Pre-trained Language Models via  Causal Inference

Non-extractive commonsense QA remains a challenging AI task, as it requires
systems to reason about, synthesize, and gather disparate pieces of
information, in order to generate responses to queries. Recent approaches on
such tasks show increased performance, only when models are either pre-trained
with additional information or when domain-specific heuristics are used,
without any special consideration regarding the knowledge resource type. In
this paper, we perform a survey of recent commonsense QA methods and we provide
a systematic analysis of popular knowledge resources and knowledge-integration
methods, across benchmarks from multiple commonsense datasets. Our results and
analysis show that attention-based injection seems to be a preferable choice
for knowledge integration and that the degree of domain overlap, between
knowledge bases and datasets, plays a crucial role in determining model
success.

本文针对非萃取式常识问答（QA）这一具有挑战性的 AI 任务进行了探讨，对最近的常识 QA 方法、流行的知识资源和知识融合方法进行了系统分析，并研究了多个常识数据集上的基准测试。结果表明，注重知识融合的注意力注入方式似乎是一种较好的选择，并且知识库与数据集之间的领域重叠度对于确定模型的成功程度具有至关重要的作用。