Counterfactual examples have proven to be valuable in the field of natural
language processing (NLP) for both evaluating and improving the robustness of
language models to spurious correlations in datasets. Despite their
demonstrated utility for NLP, multimodal counterfactual examples have been
relatively unexplored due to the difficulty of creating paired image-text data
with minimal counterfactual changes. To address this challenge, we introduce a
scalable framework for automatic generation of counterfactual examples using
text-to-image diffusion models. We use our framework to create
COCO-Counterfactuals, a multimodal counterfactual dataset of paired image and
text captions based on the MS-COCO dataset. We validate the quality of
COCO-Counterfactuals through human evaluations and show that existing
multimodal models are challenged by our counterfactual image-text pairs.
Additionally, we demonstrate the usefulness of COCO-Counterfactuals for
improving out-of-domain generalization of multimodal vision-language models via
training data augmentation.

利用文本到图像扩散模型，介绍了一种自动生成对应对的反事实例子的可扩展框架，创造了基于 MS-COCO 数据集的多模态反事实数据集 COCO-Counterfactuals，并通过人工评估验证了其质量，进一步表明现有的多模态模型受到了反事实图文对的挑战，并展示了 COCO-Counterfactuals 在通过训练数据增强改善跨域通用性的多模态视觉语言模型方面的实用性。