Vision-language pre-training methods, e.g., CLIP, demonstrate an impressive
zero-shot performance on visual categorizations with the class proxy from the
text embedding of the class name. However, the modality gap between the text
and vision space can result in a sub-optimal performance. We theoretically show
that the gap cannot be reduced sufficiently by minimizing the contrastive loss
in CLIP and the optimal proxy for vision tasks may reside only in the vision
space. Therefore, given unlabeled target vision data, we propose to learn the
vision proxy directly with the help from the text proxy for zero-shot transfer.
Moreover, according to our theoretical analysis, strategies are developed to
further refine the pseudo label obtained by the text proxy to facilitate the
intra-modal proxy learning (InMaP) for vision. Experiments on extensive
downstream tasks confirm the effectiveness and efficiency of our proposal.
Concretely, InMaP can obtain the vision proxy within one minute on a single GPU
while improving the zero-shot accuracy from $77.02\%$ to $80.21\%$ on ImageNet
with ViT-L/14@336 pre-trained by CLIP. Code is available at
https://github.com/idstcv/InMaP.

通过使用文本数据的代理标签帮助直接学习图像数据的代理标签，并利用相应的分析方法进一步提炼这些标签，以改善视觉任务中的零 - shot 分类准确率。