Fine-tuning pre-trained vision-language models, like CLIP, has yielded
success on diverse downstream tasks. However, several pain points persist for
this paradigm: (i) directly tuning entire pre-trained models becomes both
time-intensive and computationally costly. Additionally, these tuned models
tend to become highly specialized, limiting their practicality for real-world
deployment; (ii) recent studies indicate that pre-trained vision-language
classifiers may overly depend on spurious features -- patterns that correlate
with the target in training data, but are not related to the true labeling
function; and (iii) existing studies on mitigating the reliance on spurious
features, largely based on the assumption that we can identify such features,
does not provide definitive assurance for real-world applications. As a
piloting study, this work focuses on exploring mitigating the reliance on
spurious features for CLIP without using any group annotation. To this end, we
systematically study the existence of spurious correlation on CLIP and
CILP+ERM. We first, following recent work on Deep Feature Reweighting (DFR),
verify that last-layer retraining can greatly improve group robustness on
pretrained CLIP. In view of them, we advocate a lightweight representation
calibration method for fine-tuning CLIP, by first generating a calibration set
using the pretrained CLIP, and then calibrating representations of samples
within this set through contrastive learning, all without the need for group
labels. Extensive experiments and in-depth visualizations on several benchmarks
validate the effectiveness of our proposals, largely reducing reliance and
significantly boosting the model generalization.

探索在不使用任何组标注的情况下减轻 CLIP 对虚假特征依赖的方法，通过基于对比学习的轻量级表示校准方法对预训练 CLIP 进行微调，从而显著减少依赖并大大提升模型的泛化能力。

多模态表示的校准：无需标注的群体稳健性追求

Calibrating Multi-modal Representations: A Pursuit of Group Robustness  without Annotations

Predictive performance of machine learning models trained with empirical risk
minimization (ERM) can degrade considerably under distribution shifts. The
presence of spurious correlations in training datasets leads ERM-trained models
to display high loss when evaluated on minority groups not presenting such
correlations. Extensive attempts have been made to develop methods improving
worst-group robustness. However, they require group information for each
training input or at least, a validation set with group labels to tune their
hyperparameters, which may be expensive to get or unknown a priori. In this
paper, we address the challenge of improving group robustness without group
annotation during training or validation. To this end, we propose to partition
the training dataset into groups based on Gram matrices of features extracted
by an ``identification'' model and to apply robust optimization based on these
pseudo-groups. In the realistic context where no group labels are available,
our experiments show that our approach not only improves group robustness over
ERM but also outperforms all recent baselines

本文提出了一种基于 Gram 矩阵的特征提取模型对训练数据集进行分组的方法，通过伪分组采用鲁棒优化策略来提高模型的容错性能，在没有真实分组标签的情况下证明其在提高模型鲁棒性方面的优越性，并优于最近的所有基线模型。