Counterfactual image generation is pivotal for understanding the causal
relations of variables, with applications in interpretability and generation of
unbiased synthetic data. However, evaluating image generation is a
long-standing challenge in itself. The need to evaluate counterfactual
generation compounds on this challenge, precisely because counterfactuals, by
definition, are hypothetical scenarios without observable ground truths. In
this paper, we present a novel comprehensive framework aimed at benchmarking
counterfactual image generation methods. We incorporate metrics that focus on
evaluating diverse aspects of counterfactuals, such as composition,
effectiveness, minimality of interventions, and image realism. We assess the
performance of three distinct conditional image generation model types, based
on the Structural Causal Model paradigm. Our work is accompanied by a
user-friendly Python package which allows to further evaluate and benchmark
existing and future counterfactual image generation methods. Our framework is
extendable to additional SCM and other causal methods, generative models, and
datasets.

本文提出了一个综合性框架，旨在评估反事实图像生成方法，并包含了聚焦于反事实的多个方面的评估指标，如组成、有效性、干预的最小性和图像逼真度。通过基于结构因果模型范式的三种不同的条件图像生成模型类型的性能评估，证明了该框架的有效性，并提供了一个用户友好的 Python 软件包，允许进一步评估和比较现有和未来的反事实图像生成方法。该框架可扩展至其他结构因果模型和其他因果方法、生成模型和数据集。