We present a new paradigm for fine-tuning large-scale visionlanguage
pre-trained models on downstream task, dubbed Prompt Regularization (ProReg).
Different from traditional fine-tuning which easily overfits to the downstream
task data, ProReg uses the prediction by prompting the pretrained model to
regularize the fine-tuning. The motivation is: by prompting the large model "a
photo of a [CLASS]", the fil-lin answer is only dependent on the pretraining
encyclopedic knowledge while independent of the task data distribution, which
is usually biased. Specifically, given a training sample prediction during
fine-tuning, we first calculate its KullbackLeibler loss of the prompt
prediction and Cross-Entropy loss of the ground-truth label, and then combine
them with a proposed sample-wise adaptive trade-off weight, which automatically
adjusts the transfer between the pretrained and downstream domains. On various
out-of-distribution benchmarks, we show the consistently strong performance of
ProReg compared with conventional fine-tuning, zero-shot prompt, prompt tuning,
and other state-of-the-art methods.

该论文提出了一种基于提示正则化的新模式，用于在下游任务中微调大规模视觉语言预训练模型。通过提示预训练模型进行预测，来规范微调，从而保持其对迁移领域的高性能。同时，通过引入 Kullback-Leibler 损失等手段，自适应权衡预训练与下游任务训练过程中的差异，进一步提高模型性能。