Contrastive Language-Image Pre-training (CLIP) provides a foundation model by
integrating natural language into visual concepts, enabling zero-shot
recognition on downstream tasks. It is usually expected that satisfactory
overall accuracy can be achieved across numerous domains through well-designed
textual prompts. However, we found that their performance in the worst
categories is significantly inferior to the overall performance. For example,
on ImageNet, there are a total of 10 categories with class-wise accuracy as low
as 0\%, even though the overall performance has achieved 64.1\%. This
phenomenon reveals the potential risks associated with using CLIP models,
particularly in risk-sensitive applications where specific categories hold
significant importance. To address this issue, we investigate the alignment
between the two modalities in the CLIP model and propose the Class-wise
Matching Margin (\cmm) to measure the inference confusion. \cmm\ can
effectively identify the worst-performing categories and estimate the potential
performance of the candidate prompts. We further query large language models to
enrich descriptions of worst-performing categories and build a weighted
ensemble to highlight the efficient prompts. Experimental results clearly
verify the effectiveness of our proposal, where the accuracy on the worst-10
categories on ImageNet is boosted to 5.2\%, without manual prompt engineering,
laborious optimization, or access to labeled validation data.

通过研究 CLIP 模型中两种形式的对齐并提出类别匹配边界来解决其性能不足的问题，成功提高了 ImageNet 上最差 10 个类别的准确率，无需手动优化或访问标记验证数据。