Large-scale contrastive vision-language pre-trained models provide the
zero-shot model achieving competitive performance across a range of image
classification tasks without requiring training on downstream data. Recent
works have confirmed that while additional fine-tuning of the zero-shot model
on the reference data results in enhanced downstream performance, it
compromises the model's robustness against distribution shifts. Our
investigation begins by examining the conditions required to achieve the goals
of robust fine-tuning, employing descriptions based on feature distortion
theory and joint energy-based models. Subsequently, we propose a novel robust
fine-tuning algorithm, Lipsum-FT, that effectively utilizes the language
modeling aspect of the vision-language pre-trained models. Extensive
experiments conducted on distribution shift scenarios in DomainNet and ImageNet
confirm the superiority of our proposed Lipsum-FT approach over existing robust
fine-tuning methods.

大规模对比视觉 - 语言预训练模型提供了零样本模型，其可在一系列图像分类任务中取得竞争性的表现，而无需在下游数据上进行训练。最近的研究表明，对零样本模型的额外微调可以提高下游的性能，但会削弱模型对分布变化的鲁棒性。本文通过研究基于特征失真理论和联合能量模型的鲁棒微调条件，并提出一种新的鲁棒微调算法 Lipsum-FT，该算法有效地利用视觉 - 语言预训练模型中的语言建模特性。在 DomainNet 和 ImageNet 的分布变化场景上进行的大量实验实证了我们提出的 Lipsum-FT 方法优于现有的鲁棒微调方法。

Lipsum-FT：使用随机文本引导的零样本模型健壮微调

Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text  Guidance

Deploying deep visual models can lead to performance drops due to the
discrepancies between source and target distributions. Several approaches
leverage labeled source data to estimate target domain accuracy, but accessing
labeled source data is often prohibitively difficult due to data
confidentiality or resource limitations on serving devices. Our work proposes a
new framework to estimate model accuracy on unlabeled target data without
access to source data. We investigate the feasibility of using pseudo-labels
for accuracy estimation and evolve this idea into adopting recent advances in
source-free domain adaptation algorithms. Our approach measures the
disagreement rate between the source hypothesis and the target pseudo-labeling
function, adapted from the source hypothesis. We mitigate the impact of
erroneous pseudo-labels that may arise due to a high ideal joint hypothesis
risk by employing adaptive adversarial perturbation on the input of the target
model. Our proposed source-free framework effectively addresses the challenging
distribution shift scenarios and outperforms existing methods requiring source
data and labels for training.

无需访问源数据，本研究提出了一种新的框架，通过估计无标签目标数据的模型准确性来解决深度视觉模型在源和目标分布之间存在性能降低的问题。将伪标签用于准确性估计，并采用最新的无源域自适应算法，利用源假设来适应目标伪标签函数，并通过对目标模型的输入进行自适应对抗扰动以减轻错误伪标签的影响，从而有效解决具有挑战性的分布偏移情景，并胜过需要源数据和标签进行训练的现有方法。