Learning with noisy labels (LNL) has been extensively studied, with existing
approaches typically following a framework that alternates between clean sample
selection and semi-supervised learning (SSL). However, this approach has a
limitation: the clean set selected by the Deep Neural Network (DNN) classifier,
trained through self-training, inevitably contains noisy samples. This mixture
of clean and noisy samples leads to misguidance in DNN training during SSL,
resulting in impaired generalization performance due to confirmation bias
caused by error accumulation in sample selection. To address this issue, we
propose a method called Collaborative Sample Selection (CSS), which leverages
the large-scale pre-trained model CLIP. CSS aims to remove the mixed noisy
samples from the identified clean set. We achieve this by training a
2-Dimensional Gaussian Mixture Model (2D-GMM) that combines the probabilities
from CLIP with the predictions from the DNN classifier. To further enhance the
adaptation of CLIP to LNL, we introduce a co-training mechanism with a
contrastive loss in semi-supervised learning. This allows us to jointly train
the prompt of CLIP and the DNN classifier, resulting in improved feature
representation, boosted classification performance of DNNs, and reciprocal
benefits to our Collaborative Sample Selection. By incorporating auxiliary
information from CLIP and utilizing prompt fine-tuning, we effectively
eliminate noisy samples from the clean set and mitigate confirmation bias
during training. Experimental results on multiple benchmark datasets
demonstrate the effectiveness of our proposed method in comparison with the
state-of-the-art approaches.

利用 CLIP 模型的协作样本选择及预训练，并通过对 prompt 的微调以及协同训练 DNN 分类器，解决在学习有噪声标签的过程中由于样本选择错误累积导致的 DNN 训练偏见和泛化性能问题。