Contrastive language-image pre-training (CLIP) has demonstrated remarkable
zero-shot classification ability, namely image classification using novel text
labels. Existing works have attempted to enhance CLIP by fine-tuning on
downstream tasks, but these have inadvertently led to performance degradation
on unseen classes, thus harming zero-shot generalization. This paper aims to
address this challenge by leveraging readily available image-text pairs from an
external dataset for cross-modal guidance during inference. To this end, we
propose X-MoRe, a novel inference method comprising two key steps: (1)
cross-modal retrieval and (2) modal-confidence-based ensemble. Given a query
image, we harness the power of CLIP's cross-modal representations to retrieve
relevant textual information from an external image-text pair dataset. Then, we
assign higher weights to the more reliable modality between the original query
image and retrieved text, contributing to the final prediction. X-MoRe
demonstrates robust performance across a diverse set of tasks without the need
for additional training, showcasing the effectiveness of utilizing cross-modal
features to maximize CLIP's zero-shot ability.

通过跨模态引导和模态置信度集成，X-MoRe 方法利用 CLIP 的跨模态表示能力，从外部图文对数据集中检索相关的文本信息，并通过赋予可靠性更高的模态对最终预测产生贡献，从而在多样化的任务中展示了稳健的性能，充分发挥了 CLIP 的零样本分类能力。