Pre-trained large-scale vision-language models (VLMs) have acquired profound
understanding of general visual concepts. Recent advancements in efficient
transfer learning (ETL) have shown remarkable success in fine-tuning VLMs
within the scenario of limited data, introducing only a few parameters to
harness task-specific insights from VLMs. Despite significant progress, current
leading ETL methods tend to overfit the narrow distributions of base classes
seen during training and encounter two primary challenges: (i) only utilizing
uni-modal information to modeling task-specific knowledge; and (ii) using
costly and time-consuming methods to supplement knowledge. To address these
issues, we propose a Conditional Prototype Rectification Prompt Learning (CPR)
method to correct the bias of base examples and augment limited data in an
effective way. Specifically, we alleviate overfitting on base classes from two
aspects. First, each input image acquires knowledge from both textual and
visual prototypes, and then generates sample-conditional text tokens. Second,
we extract utilizable knowledge from unlabeled data to further refine the
prototypes. These two strategies mitigate biases stemming from base classes,
yielding a more effective classifier. Extensive experiments on 11 benchmark
datasets show that our CPR achieves state-of-the-art performance on both
few-shot classification and base-to-new generalization tasks. Our code is
avaliable at https://github.com/chenhaoxing/CPR.

利用大规模的预先训练的视觉语言模型，通过高效的迁移学习方法解决基类过拟合问题，提出了一种条件原型纠正提示学习方法，有效地纠正了基类示例的偏差并增加了有限数据量，在少样本分类和从基类到新类的泛化任务中实现了最先进的性能。

条件下的原型修正提示学习

Conditional Prototype Rectification Prompt Learning

Efficient transfer learning (ETL) is receiving increasing attention to adapt
large pre-trained language-vision models on downstream tasks with a few labeled
samples. While significant progress has been made, we reveal that
state-of-the-art ETL approaches exhibit strong performance only in
narrowly-defined experimental setups, and with a careful adjustment of
hyperparameters based on a large corpus of labeled samples. In particular, we
make two interesting, and surprising empirical observations. First, to
outperform a simple Linear Probing baseline, these methods require to optimize
their hyper-parameters on each target task. And second, they typically
underperform -- sometimes dramatically -- standard zero-shot predictions in the
presence of distributional drifts. Motivated by the unrealistic assumptions
made in the existing literature, i.e., access to a large validation set and
case-specific grid-search for optimal hyperparameters, we propose a novel
approach that meets the requirements of real-world scenarios. More concretely,
we introduce a CLass-Adaptive linear Probe (CLAP) objective, whose balancing
term is optimized via an adaptation of the general Augmented Lagrangian method
tailored to this context. We comprehensively evaluate CLAP on a broad span of
datasets and scenarios, demonstrating that it consistently outperforms SoTA
approaches, while yet being a much more efficient alternative.

通过引入适应真实场景需求的新方法，我们综合评估了一个广泛的数据集和场景，发现其在实践中始终优于现有技术，同时作为更高效的替代方案。

大型视觉语言模型的少样本自适应研究

A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models

Adapter-style efficient transfer learning (ETL) has shown excellent
performance in the tuning of vision-language models (VLMs) under the low-data
regime, where only a few additional parameters are introduced to excavate the
task-specific knowledge based on the general and powerful representation of
VLMs. However, most adapter-style works face two limitations: (i) modeling
task-specific knowledge with a single modality only; and (ii) overlooking the
exploitation of the inter-class relationships in downstream tasks, thereby
leading to sub-optimal solutions. To mitigate that, we propose an effective
adapter-style tuning strategy, dubbed GraphAdapter, which performs the textual
adapter by explicitly modeling the dual-modality structure knowledge (i.e., the
correlation of different semantics/classes in textual and visual modalities)
with a dual knowledge graph. In particular, the dual knowledge graph is
established with two sub-graphs, i.e., a textual knowledge sub-graph, and a
visual knowledge sub-graph, where the nodes and edges represent the
semantics/classes and their correlations in two modalities, respectively. This
enables the textual feature of each prompt to leverage the task-specific
structure knowledge from both textual and visual modalities, yielding a more
effective classifier for downstream tasks. Extensive experimental results on 11
benchmark datasets reveal that our GraphAdapter significantly outperforms
previous adapter-based methods. The code will be released at
this https URL

通过建立双重知识图，将文本和视觉语义 / 类别之间的关联性进行建模，利用适配器样式的调整策略，在少量数据条件下对视觉 - 语言模型进行调优，提高下游任务的分类器的效果。

GraphAdapter: 用双重知识图调整视觉语言模型

GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph

Large-scale vision-language models (VLMs) pre-trained on billion-level data
have learned general visual representations and broad visual concepts. In
principle, the well-learned knowledge structure of the VLMs should be inherited
appropriately when being transferred to downstream tasks with limited data.
However, most existing efficient transfer learning (ETL) approaches for VLMs
either damage or are excessively biased towards the prior knowledge, e.g.,
prompt tuning (PT) discards the pre-trained text-based classifier and builds a
new one while adapter-style tuning (AT) fully relies on the pre-trained
features. To address this, we propose a new efficient tuning approach for VLMs
named Task Residual Tuning (TaskRes), which performs directly on the text-based
classifier and explicitly decouples the prior knowledge of the pre-trained
models and new knowledge regarding a target task. Specifically, TaskRes keeps
the original classifier weights from the VLMs frozen and obtains a new
classifier for the target task by tuning a set of prior-independent parameters
as a residual to the original one, which enables reliable prior knowledge
preservation and flexible task-specific knowledge exploration. The proposed
TaskRes is simple yet effective, which significantly outperforms previous ETL
methods (e.g., PT and AT) on 11 benchmark datasets while requiring minimal
effort for the implementation. Our code is available at
this https URL

本篇论文提出了一种名为 Task Residual Tuning 的新的视觉语言模型的高效调整方法，其中通过保留原有分类器权重的方式，针对目标任务调整一组先独立参数作为原有分类器的残差，以实现可靠的先验知识保存和灵活的任务特定知识探索，并在 11 个基准数据集中显著优于当前的高效转移学习方法。