Efficient transfer learning (ETL) is receiving increasing attention to adapt
large pre-trained language-vision models on downstream tasks with a few labeled
samples. While significant progress has been made, we reveal that
state-of-the-art ETL approaches exhibit strong performance only in
narrowly-defined experimental setups, and with a careful adjustment of
hyperparameters based on a large corpus of labeled samples. In particular, we
make two interesting, and surprising empirical observations. First, to
outperform a simple Linear Probing baseline, these methods require to optimize
their hyper-parameters on each target task. And second, they typically
underperform -- sometimes dramatically -- standard zero-shot predictions in the
presence of distributional drifts. Motivated by the unrealistic assumptions
made in the existing literature, i.e., access to a large validation set and
case-specific grid-search for optimal hyperparameters, we propose a novel
approach that meets the requirements of real-world scenarios. More concretely,
we introduce a CLass-Adaptive linear Probe (CLAP) objective, whose balancing
term is optimized via an adaptation of the general Augmented Lagrangian method
tailored to this context. We comprehensively evaluate CLAP on a broad span of
datasets and scenarios, demonstrating that it consistently outperforms SoTA
approaches, while yet being a much more efficient alternative.

通过引入适应真实场景需求的新方法，我们综合评估了一个广泛的数据集和场景，发现其在实践中始终优于现有技术，同时作为更高效的替代方案。

大型视觉语言模型的少样本自适应研究

A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models

Pre-trained language-vision models have shown remarkable performance on the
visual question answering (VQA) task. However, most pre-trained models are
trained by only considering monolingual learning, especially the resource-rich
language like English. Training such models for multilingual setups demand high
computing resources and multilingual language-vision dataset which hinders
their application in practice. To alleviate these challenges, we propose a
knowledge distillation approach to extend an English language-vision model
(teacher) into an equally effective multilingual and code-mixed model
(student). Unlike the existing knowledge distillation methods, which only use
the output from the last layer of the teacher network for distillation, our
student model learns and imitates the teacher from multiple intermediate layers
(language and vision encoders) with appropriately designed distillation
objectives for incremental knowledge extraction. We also create the large-scale
multilingual and code-mixed VQA dataset in eleven different language setups
considering the multiple Indian and European languages. Experimental results
and in-depth analysis show the effectiveness of the proposed VQA model over the
pre-trained language-vision models on eleven diverse language setups.

提出一种基于知识蒸馏方法的英语语言 - 视觉模型拓展到同等效果的多语种和混合编码模型的方法，其使用多个中间层来逐步提取知识，同时在 11 种不同语言设置下创建大规模的多语种和混合编码的 VQA 数据集。