The state-of-the-art pre-trained language representation models, such as
Bidirectional Encoder Representations from Transformers (BERT), rarely
incorporate commonsense knowledge or other knowledge explicitly. We propose a
pre-training approach for incorporating commonsense knowledge into language
representation models. We construct a commonsense-related multi-choice question
answering dataset for pre-training a neural language representation model. The
dataset is created automatically by our proposed "align, mask, and select"
(AMS) method. We also investigate different pre-training tasks. Experimental
results demonstrate that pre-training models using the proposed approach
followed by fine-tuning achieve significant improvements over previous
state-of-the-art models on two commonsense-related benchmarks, including
CommonsenseQA and Winograd Schema Challenge. We also observe that fine-tuned
models after the proposed pre-training approach maintain comparable performance
on other NLP tasks, such as sentence classification and natural language
inference tasks, compared to the original BERT models. These results verify
that the proposed approach, while significantly improving commonsense-related
NLP tasks, does not degrade the general language representation capabilities.

提出了一种利用预训练方法将常识知识融入语言表示模型的方法，并通过训练一个多项选择问题回答数据集来实现。实验结果表明，使用该方法进行预训练之后 Fine-tuning 的模型，在两个常识关联基准测试上都能取得显著提高，并且不会降低模型的语言表示能力。