Previous studies have shown the efficacy of knowledge augmentation methods in
pretrained language models. However, these methods behave differently across
domains and downstream tasks. In this work, we investigate the augmentation of
pretrained language models with commonsense knowledge in the cause-effect
relation classification and commonsense causal reasoning tasks. After
automatically verbalizing ATOMIC2020, a wide coverage commonsense reasoning
knowledge graph, and GLUCOSE, a dataset of implicit commonsense causal
knowledge, we continually pretrain BERT and RoBERTa with the verbalized data.
Then we evaluate the resulting models on cause-effect pair classification and
answering commonsense causal reasoning questions. Our results show that
continually pretrained language models augmented with commonsense knowledge
outperform our baselines on two commonsense causal reasoning benchmarks, COPA
and BCOPA-CE, and the Temporal and Causal Reasoning (TCR) dataset, without
additional improvement in model architecture or using quality-enhanced data for
fine-tuning.

本文研究如何将常识知识加入预训练的语言模型中，并发现这种方式可以提高模型在因果关系分类和常识因果推理任务中的表现。