Large-scale contrastive vision-language pre-trained models provide the
zero-shot model achieving competitive performance across a range of image
classification tasks without requiring training on downstream data. Recent
works have confirmed that while additional fine-tuning of the zero-shot model
on the reference data results in enhanced downstream performance, it
compromises the model's robustness against distribution shifts. Our
investigation begins by examining the conditions required to achieve the goals
of robust fine-tuning, employing descriptions based on feature distortion
theory and joint energy-based models. Subsequently, we propose a novel robust
fine-tuning algorithm, Lipsum-FT, that effectively utilizes the language
modeling aspect of the vision-language pre-trained models. Extensive
experiments conducted on distribution shift scenarios in DomainNet and ImageNet
confirm the superiority of our proposed Lipsum-FT approach over existing robust
fine-tuning methods.

大规模对比视觉 - 语言预训练模型提供了零样本模型，其可在一系列图像分类任务中取得竞争性的表现，而无需在下游数据上进行训练。最近的研究表明，对零样本模型的额外微调可以提高下游的性能，但会削弱模型对分布变化的鲁棒性。本文通过研究基于特征失真理论和联合能量模型的鲁棒微调条件，并提出一种新的鲁棒微调算法 Lipsum-FT，该算法有效地利用视觉 - 语言预训练模型中的语言建模特性。在 DomainNet 和 ImageNet 的分布变化场景上进行的大量实验实证了我们提出的 Lipsum-FT 方法优于现有的鲁棒微调方法。