Pre-trained vision-language models (VLMs), exemplified by CLIP, demonstrate
remarkable adaptability across zero-shot classification tasks without
additional training. However, their performance diminishes in the presence of
domain shifts. In this study, we introduce CLIP Adaptation duRing Test-Time
(CLIPArTT), a fully test-time adaptation (TTA) approach for CLIP, which
involves automatic text prompts construction during inference for their use as
text supervision. Our method employs a unique, minimally invasive text prompt
tuning process, wherein multiple predicted classes are aggregated into a single
new text prompt, used as pseudo label to re-classify inputs in a transductive
manner. Additionally, we pioneer the standardization of TTA benchmarks (e.g.,
TENT) in the realm of VLMs. Our findings demonstrate that, without requiring
additional transformations nor new trainable modules, CLIPArTT enhances
performance dynamically across non-corrupted datasets such as CIFAR-10,
corrupted datasets like CIFAR-10-C and CIFAR-10.1, alongside synthetic datasets
such as VisDA-C. This research underscores the potential for improving VLMs'
adaptability through novel test-time strategies, offering insights for robust
performance across varied datasets and environments. The code can be found at:
this https URL

通过在推理过程中自动构建文本提示并使用其作为文本监督，CLIPArTT 方法在不需要额外培训的情况下，通过独特的、最小侵入性的文本提示调优过程，使预训练视觉 - 语言模型在各种数据集和环境中动态提升性能并改善适应性。