In the realm of deep learning, maintaining model robustness against
distribution shifts is critical. This paper investigates test-time adaptation
strategies for vision-language models, with a specific focus on CLIP and its
variants. Through a systematic exploration of prompt-based techniques and
existing test-time adaptation methods, the study aims to enhance the
adaptability and robustness of vision-language models in diverse real-world
scenarios. The investigation includes an analysis of prompt engineering
strategies, such as hand-crafted prompts, prompt ensembles, and prompt learning
techniques. We introduce a vision-text-space ensemble that significantly boosts
the average performance compared to a text-space-only ensemble. Additionally,
our comparative study delves into leveraging existing test-time adaptation
methods originally designed for image classification tasks. Experimental
evaluations conducted across various datasets and model architectures
demonstrate the efficacy of different adaptation strategies. We further give
insights into the importance of updating the vision encoder and whether it is
beneficial to update the text encoder. Code is available at
this https URL

对视觉 - 语言模型的测试时间适应性策略进行系统探索，包括提示工程策略和现有的测试时间适应方法，以提高模型在真实场景中的适应性和鲁棒性。

视觉语言模型的失误机会：视觉语言模型在线测试时间适应的比较研究

A Lost Opportunity for Vision-Language Models: A Comparative Study of  Online Test-time Adaptation for Vision-Language Models

Training state-of-the-art vision models has become prohibitively expensive
for researchers and practitioners. For the sake of accessibility and resource
reuse, it is important to focus on adapting these models to a variety of
downstream scenarios. An interesting and practical paradigm is online test-time
adaptation, according to which training data is inaccessible, no labelled data
from the test distribution is available, and adaptation can only happen at test
time and on a handful of samples. In this paper, we investigate how test-time
adaptation methods fare for a number of pre-trained models on a variety of
real-world scenarios, significantly extending the way they have been originally
evaluated. We show that they perform well only in narrowly-defined experimental
setups and sometimes fail catastrophically when their hyperparameters are not
selected for the same scenario in which they are being tested. Motivated by the
inherent uncertainty around the conditions that will ultimately be encountered
at test time, we propose a particularly "conservative" approach, which
addresses the problem with a Laplacian Adjusted Maximum-likelihood Estimation
(LAME) objective. By adapting the model's output (not its parameters), and
solving our objective with an efficient concave-convex procedure, our approach
exhibits a much higher average accuracy across scenarios than existing methods,
while being notably faster and have a much lower memory footprint. The code is
available at this https URL

为了使现有的视觉模型适应于不同的下游场景，本文探讨了在线测试时间自适应方法在各种真实世界场景下的表现，并针对该方法的局限性提出了一种特别保守的方法 —— 使用 LAME 目标来解决问题，并通过高效的凸凹过程求解该目标，从而使其在场景中表现更佳。