The role of data in building AI systems has recently been emphasized by the
emerging concept of data-centric AI. Unfortunately, in the real-world, datasets
may contain dirty samples, such as poisoned samples from backdoor attack, noisy
labels in crowdsourcing, and even hybrids of them. The presence of such dirty
samples makes the DNNs vunerable and unreliable.Hence, it is critical to detect
dirty samples to improve the quality and realiability of dataset. Existing
detectors only focus on detecting poisoned samples or noisy labels, that are
often prone to weak generalization when dealing with dirty samples from other
domains.In this paper, we find a commonality of various dirty samples is
visual-linguistic inconsistency between images and associated labels. To
capture the semantic inconsistency between modalities, we propose versatile
data cleanser (VDC) leveraging the surpassing capabilities of multimodal large
language models (MLLM) in cross-modal alignment and reasoning.It consists of
three consecutive modules: the visual question generation module to generate
insightful questions about the image; the visual question answering module to
acquire the semantics of the visual content by answering the questions with
MLLM; followed by the visual answer evaluation module to evaluate the
inconsistency.Extensive experiments demonstrate its superior performance and
generalization to various categories and types of dirty samples.

通过使用多模态的大型语言模型，本研究提出了一种名为 VDC 的多功能数据清洗器，用于检测数据集中的污染样本，特别关注图像和相关标签之间的语义不一致性，实验证明其在各种类型的污染样本上具有卓越的性能和推广能力。