Latest methods for visual counterfactual explanations (VCE) harness the power of deep generative models to synthesize new examples of high-dimensional images of impressive quality. However, it is currently difficult to compare the performance of these VCE methods as the evaluation procedures largely vary and often boil down to visual inspection of individual examples and small scale user studies. In this work, we propose a framework for systematic, quantitative evaluation of the VCE methods and a minimal set of metrics to be used. We use this framework to explore the effects of certain crucial design choices in the latest diffusion-based generative models for VCEs of natural image classification (ImageNet). We conduct a battery of ablation-like experiments, generating thousands of VCEs for a suite of classifiers of various complexity, accuracy and robustness. Our findings suggest multiple directions for future advancements and improvements of VCE methods. By sharing our methodology and our approach to tackle the computational challenges of such a study on a limited hardware setup (including the complete code base), we offer a valuable guidance for researchers in the field fostering consistency and transparency in the assessment of counterfactual explanations.

最新的视觉反事实解释方法利用深度生成模型的能力合成了高维度图像的新示例，本研究提出了一个系统的、量化的评估框架以及一组最小的指标，探索了最新的基于扩散的生成模型在自然图像分类的反事实解释方面的关键设计选择的效果，并通过产生数千个反事实解释来对各种复杂度、准确性和鲁棒性的分类器进行了一系列消融实验，为未来反事实解释方法的进一步改进和发展提出了多个方向，通过分享研究方法和解决计算挑战的方法，为该领域的研究人员提供了宝贵的指导，促进了对反事实解释的评估的一致性和透明度。

基于扩散的视觉反事实解释——朝向系统化定量评估