Although vision models such as Contrastive Language-Image Pre-Training (CLIP)
show impressive generalization performance, their zero-shot robustness is still
limited under Out-of-Distribution (OOD) scenarios without fine-tuning. Instead
of undesirably providing human supervision as commonly done, it is possible to
take advantage of Multi-modal Large Language Models (MLLMs) that hold powerful
visual understanding abilities. However, MLLMs are shown to struggle with
vision problems due to the incompatibility of tasks, thus hindering their
utilization. In this paper, we propose to effectively leverage MLLMs to conduct
Machine Vision Therapy which aims to rectify the noisy predictions from vision
models. By fine-tuning with the denoised labels, the learning model performance
can be boosted in an unsupervised manner. To solve the incompatibility issue,
we propose a novel Denoising In-Context Learning (DICL) strategy to align
vision tasks with MLLMs. Concretely, by estimating a transition matrix that
captures the probability of one class being confused with another, an
instruction containing a correct exemplar and an erroneous one from the most
probable noisy class can be constructed. Such an instruction can help any MLLMs
with ICL ability to detect and rectify incorrect predictions of vision models.
Through extensive experiments on ImageNet, WILDS, DomainBed, and other OOD
datasets, we carefully validate the quantitative and qualitative effectiveness
of our method. Our code is available at
this https URL

我们提出了一种有效利用 MLLMs 进行机器视觉治疗的方法，通过与去噪标签进行微调，以无监督的方式提高学习模型的性能，并通过提出的 DICL 策略解决了 MLLMs 与视觉任务的兼容性问题。