Pre-trained vision-language models have shown impressive success on various
computer vision tasks with their zero-shot generalizability. Recently, prompt
learning approaches have been explored to efficiently and effectively adapt the
vision-language models to a variety of downstream tasks. However, most existing
prompt learning methods suffer from task overfitting since the general
knowledge of the pre-trained vision language models is forgotten while the
prompts are finetuned on a small data set from a specific target task. To
address this issue, we propose a Prompt Meta-Regularization (ProMetaR) to
improve the generalizability of prompt learning for vision-language models.
Specifically, ProMetaR meta-learns both the regularizer and the soft prompts to
harness the task-specific knowledge from the downstream tasks and task-agnostic
general knowledge from the vision-language models. Further, ProMetaR augments
the task to generate multiple virtual tasks to alleviate the meta-overfitting.
In addition, we provide the analysis to comprehend how ProMetaR improves the
generalizability of prompt tuning in the perspective of the gradient alignment.
Our extensive experiments demonstrate that our ProMetaR improves the
generalizability of conventional prompt learning methods under
base-to-base/base-to-new and domain generalization settings. The code of
ProMetaR is available at this https URL

通过 Prompt Meta-Regularization (ProMetaR) 方法，可以提高视觉语言模型中 prompt 学习的泛化能力，并可以从梯度对齐的角度解释这种改进。

元规范化的提示学习

Prompt Learning via Meta-Regularization

Meta learning recently has been heavily researched and helped advance the
contemporary machine learning. However, achieving well-performing meta-learning
model requires a large amount of training tasks with high-quality meta-data
representing the underlying task generalization goal, which is sometimes
difficult and expensive to obtain for real applications. Current
meta-data-driven meta-learning approaches, however, are fairly hard to train
satisfactory meta-models with imperfect training tasks. To address this issue,
we suggest a meta-knowledge informed meta-learning (MKIML) framework to improve
meta-learning by additionally integrating compensated meta-knowledge into
meta-learning process. We preliminarily integrate meta-knowledge into
meta-objective via using an appropriate meta-regularization (MR) objective to
regularize capacity complexity of the meta-model function class to facilitate
better generalization on unseen tasks. As a practical implementation, we
introduce data augmentation consistency to encode invariance as meta-knowledge
for instantiating MR objective, denoted by DAC-MR. The proposed DAC-MR is
hopeful to learn well-performing meta-models from training tasks with noisy,
sparse or unavailable meta-data. We theoretically demonstrate that DAC-MR can
be treated as a proxy meta-objective used to evaluate meta-model without
high-quality meta-data. Besides, meta-data-driven meta-loss objective combined
with DAC-MR is capable of achieving better meta-level generalization. 10
meta-learning tasks with different network architectures and benchmarks
substantiate the capability of our DAC-MR on aiding meta-model learning. Fine
performance of DAC-MR are obtained across all settings, and are well-aligned
with our theoretical insights. This implies that our DAC-MR is
problem-agnostic, and hopeful to be readily applied to extensive meta-learning
problems and tasks.

在元学习中应用元知识来改善元学习的效果，并提出了基于数据扩充一致性的元知识来进行元正则化，实现无噪音、稀疏或无法使用高质量元数据的训练任务的元模型学习，从而在元层面上实现更好的泛化。