Vision models with high overall accuracy often exhibit systematic errors in
specific scenarios, posing potential serious safety concerns. Diagnosing bugs
of vision models is gaining increased attention, however traditional diagnostic
approaches require annotation efforts (\eg rich metadata accompanying each
samples of CelebA). To address this issue,We propose a language-assisted
diagnostic method that uses texts instead of images to diagnose bugs in vision
models based on multi-modal models (\eg CLIP). Our approach connects the
embedding space of CLIP with the buggy vision model to be diagnosed; meanwhile,
utilizing a shared classifier and the cross-modal transferability of embedding
space from CLIP, the text-branch of CLIP become a proxy model to find bugs in
the buggy model. The proxy model can classify texts paired with images. During
the diagnosis, a Large Language Model (LLM) is employed to obtain task-relevant
corpora, and this corpora is used to extract keywords. Descriptions constructed
with templates containing these keywords serve as input text to probe errors in
the proxy model. Finally, we validate the ability to diagnose existing visual
models using language on the Waterbirds and CelebA datasets, we can identify
bugs comprehensible to human experts, uncovering not only known bugs but also
previously unknown ones.

通过语言辅助诊断方法，使用文本而非图像来诊断视觉模型中的错误，通过嵌入空间和共享分类器及跨模态迁移识别图像，验证了在水鸟和 CelebA 数据集上诊断现有视觉模型的能力。