The perturbation test remains the go-to evaluation approach for explanation
methods in computer vision. This evaluation method has a major drawback of
test-time distribution shift due to pixel-masking that is not present in the
training set. To overcome this drawback, we propose a novel evaluation
framework called \textbf{Inpainting the Gaps (InG)}. Specifically, we propose
inpainting parts that constitute partial or complete objects in an image. In
this way, one can perform meaningful image perturbations with lower test-time
distribution shifts, thereby improving the efficacy of the perturbation test.
InG is applied to the PartImageNet dataset to evaluate the performance of
popular explanation methods for three training strategies of the Vision
Transformer (ViT). Based on this evaluation, we found Beyond Intuition and
Generic Attribution to be the two most consistent explanation models. Further,
and interestingly, the proposed framework results in higher and more consistent
evaluation scores across all the ViT models considered in this work. To the
best of our knowledge, InG is the first semi-synthetic framework for the
evaluation of ViT explanation methods.

借助 “Inpainting the Gaps（InG）” 这一创新的评估框架，针对计算机视觉领域中的解释方法，本研究通过修复图像中的空缺部分，降低测试时间分布漂移，提高了扰动测试的效果，并在 Vision Transformer（ViT）的三种训练策略上进行了 PartImageNet 数据集的评估，发现 Beyond Intuition 和 Generic Attribution 是最一致的解释模型，此外，该框架还在所有考虑的 ViT 模型中得到了更高和更一致的评估分数。在我们所了解的范围内，InG 是评估 ViT 解释方法的第一个半合成框架。