A common approach to explain NLP models, is to use importance measures that
express which tokens are important for a prediction. Unfortunately, such
explanations are often wrong despite being persuasive. Therefore, it is
essential to measure their faithfulness. One such metric is if tokens are truly
important, then masking them should result in worse model performance. However,
token masking introduces out-of-distribution issues and existing solutions are
computationally expensive and employ proxy-models. Furthermore, other metrics
are very limited in scope. In this work, we propose an inherently faithfulness
measurable model that addresses these challenges. This is achieved by using a
novel fine-tuning method that incorporates masking, such that masking tokens
become in-distribution by design. This differs from existing approaches, which
are completely model-agnostic but are inapplicable in practice. We demonstrate
the generality of our approach by applying it to various tasks and validate it
using statistical in-distribution tests. Additionally, because masking is
in-distribution, importance measures which themselves use masking become more
faithful, thus our model becomes more explainable.

提议基于掩码方法的一种内在的可靠度可衡量模型，该模型能够解决现有方法受限、计算昂贵且基于代理模型的问题，通过在设计上使掩码令牌成为数据分布内的方式，从而能够更有效地解释和说明 NLP 模型。

可衡量忠诚度的遮蔽语言模型

Faithfulness Measurable Masked Language Models

While large code datasets have become available in recent years, acquiring
representative training data with full coverage of general code distribution
remains challenging due to the compositional nature of code and the complexity
of software. This leads to the out-of-distribution (OOD) issues with unexpected
model inference behaviors that have not been systematically studied yet. We
contribute the first systematic approach that simulates various OOD scenarios
along different dimensions of data properties and investigates the model
behaviors in such scenarios. Our extensive studies on six state-of-the-art
models for three code generation tasks expose several failure modes caused by
the out-of-distribution issues. It thereby provides insights and sheds light
for future research in terms of generalization, robustness, and inductive
biases of source code models.

由于代码的组合性和软件的复杂性，获取具有完整代码分布覆盖的代表性训练数据仍然具有挑战性。因此，我们提出了第一种系统性方法，模拟不同维度数据属性的各种 OOD 场景，并研究在这些场景下模型的行为。我们对三项代码生成任务中的六种最先进的模型进行了广泛的研究，发现了一些由 OOD 问题引起的失败模式。此研究具有启示作用，为未来关于源代码模型的泛化、鲁棒性和归纳偏差等的研究提供了新的视角。