This paper proposes a novel framework to reinforce classification models
using language-guided generated counterfactual images. Deep learning
classification models are often trained using datasets that mirror real-world
scenarios. In this training process, because learning is based solely on
correlations with labels, there is a risk that models may learn spurious
relationships, such as an overreliance on features not central to the subject,
like background elements in images. However, due to the black-box nature of the
decision-making process in deep learning models, identifying and addressing
these vulnerabilities has been particularly challenging. We introduce a novel
framework for reinforcing the classification models, which consists of a
two-stage process. First, we identify model weaknesses by testing the model
using the counterfactual image dataset, which is generated by perturbed image
captions. Subsequently, we employ the counterfactual images as an augmented
dataset to fine-tune and reinforce the classification model. Through extensive
experiments on several classification models across various datasets, we
revealed that fine-tuning with a small set of counterfactual images effectively
strengthens the model.

该论文提出了一种新的框架，通过语言引导生成对抗图片来加强分类模型。通过使用对抗图片数据集来测试模型的弱点，并将对抗图片作为增加的数据集来微调和加固分类模型，研究揭示了使用小规模对抗图片进行微调可以有效增强模型的性能。