While pre-trained language models (LMs) have brought great improvements in
many NLP tasks, there is increasing attention to explore capabilities of LMs
and interpret their predictions. However, existing works usually focus only on
a certain capability with some downstream tasks. There is a lack of datasets
for directly evaluating the masked word prediction performance and the
interpretability of pre-trained LMs. To fill in the gap, we propose a novel
evaluation benchmark providing with both English and Chinese annotated data. It
tests LMs abilities in multiple dimensions, i.e., grammar, semantics,
knowledge, reasoning and computation. In addition, it provides carefully
annotated token-level rationales that satisfy sufficiency and compactness. It
contains perturbed instances for each original instance, so as to use the
rationale consistency under perturbations as the metric for faithfulness, a
perspective of interpretability. We conduct experiments on several widely-used
pre-trained LMs. The results show that they perform very poorly on the
dimensions of knowledge and computation. And their plausibility in all
dimensions is far from satisfactory, especially when the rationale is short. In
addition, the pre-trained LMs we evaluated are not robust on syntax-aware data.
We will release this evaluation benchmark at http://xyz, and hope it can
facilitate the research progress of pre-trained LMs.

本文提出一种基于英文与中文标注数据的新型评估基准，旨在测试预训练语言模型在语法、语义、知识、推理和计算等多个方面的能力，并提供满足充分性和紧凑性的标注标记级别的理由，以及人性化的实例扰动，从而使用照常理性度量：诚实度的角度。实验结果表明，预训练语言模型在知识和计算方面表现非常糟糕；尤其是在理由较短的情况下，其所有维度上的可信度都远远不足够，并且在语法感知数据上评估的预训练语言模型不稳健。