Contrastive Language-Image Pre-training (CLIP) has been shown to learn visual
representations with great transferability, which achieves promising accuracy
for zero-shot classification. To further improve its downstream performance,
existing works propose additional learnable modules upon CLIP and fine-tune
them by few-shot training sets. However, the resulting extra training cost and
data requirement severely hinder the efficiency for model deployment and
knowledge transfer. In this paper, we introduce a free-lunch enhancement
method, CALIP, to boost CLIP's zero-shot performance via a parameter-free
Attention module. Specifically, we guide visual and textual representations to
interact with each other and explore cross-modal informative features via
attention. As the pre-training has largely reduced the embedding distances
between two modalities, we discard all learnable parameters in the attention
and bidirectionally update the multi-modal features, enabling the whole process
to be parameter-free and training-free. In this way, the images are blended
with textual-aware signals and the text representations become visual-guided
for better adaptive zero-shot alignment. We evaluate CALIP on various
benchmarks of 14 datasets for both 2D image and 3D point cloud few-shot
classification, showing consistent zero-shot performance improvement over CLIP.
Based on that, we further insert a small number of linear layers in CALIP's
attention module and verify our robustness under the few-shot settings, which
also achieves leading performance compared to existing methods. Those extensive
experiments demonstrate the superiority of our approach for efficient
enhancement of CLIP.

本文提出了一种名为 CALIP 的方法，通过一个无需参数的关注模块，在不增加额外的训练开销和数据需求的情况下提高 Contrastive Language-Image Pre-training 的零样本性能。