Many contextualized word representations are now learned by intricate neural
network models, such as masked neural language models (MNLMs) which are made up
of huge neural network structures and trained to restore the masked text. Such
representations demonstrate superhuman performance in some reading
comprehension (RC) tasks which extract a proper answer in the context given a
question. However, identifying the detailed knowledge trained in MNLMs is
challenging owing to numerous and intermingled model parameters. This paper
provides new insights and empirical analyses on commonsense knowledge included
in pretrained MNLMs. First, we use a diagnostic test that evaluates whether
commonsense knowledge is properly trained in MNLMs. We observe that a large
proportion of commonsense knowledge is not appropriately trained in MNLMs and
MNLMs do not often understand the semantic meaning of relations accurately. In
addition, we find that the MNLM-based RC models are still vulnerable to
semantic variations that require commonsense knowledge. Finally, we discover
the fundamental reason why some knowledge is not trained. We further suggest
that utilizing an external commonsense knowledge repository can be an effective
solution. We exemplify the possibility to overcome the limitations of the
MNLM-based RC models by enriching text with the required knowledge from an
external commonsense knowledge repository in controlled experiments.

本文通过使用诊断测试，揭示了预训练 MNLM 中包含的常识知识，并发现 MNLM 模型不能准确理解语义关系，还存在对需要常识知识的语义变化的脆弱性。此外，我们发现了某些知识未被训练的根本原因，并通过在受控实验中利用外部常识知识库来丰富文本的方法，提出了克服 MNLM 模型的局限性的可能性。