How do two sets of images differ? Discerning set-level differences is crucial
for understanding model behaviors and analyzing datasets, yet manually sifting
through thousands of images is impractical. To aid in this discovery process,
we explore the task of automatically describing the differences between two
$\textbf{sets}$ of images, which we term Set Difference Captioning. This task
takes in image sets $D_A$ and $D_B$, and outputs a description that is more
often true on $D_A$ than $D_B$. We outline a two-stage approach that first
proposes candidate difference descriptions from image sets and then re-ranks
the candidates by checking how well they can differentiate the two sets. We
introduce VisDiff, which first captions the images and prompts a language model
to propose candidate descriptions, then re-ranks these descriptions using CLIP.
To evaluate VisDiff, we collect VisDiffBench, a dataset with 187 paired image
sets with ground truth difference descriptions. We apply VisDiff to various
domains, such as comparing datasets (e.g., ImageNet vs. ImageNetV2), comparing
classification models (e.g., zero-shot CLIP vs. supervised ResNet), summarizing
model failure modes (supervised ResNet), characterizing differences between
generative models (e.g., StableDiffusionV1 and V2), and discovering what makes
images memorable. Using VisDiff, we are able to find interesting and previously
unknown differences in datasets and models, demonstrating its utility in
revealing nuanced insights.

自动描述两个图像集之间的差异，并通过对候选描述进行重新排序来区分两个集合，从而揭示数据集和模型中的潜在差异