Contrastive Language-Image Pre-training (CLIP) is a powerful multimodal large
vision model that has demonstrated significant benefits for downstream tasks,
including many zero-shot learning and text-guided vision tasks. However, we
notice some severe problems regarding the model's explainability, which
undermines its credibility and impedes related tasks. Specifically, we find
CLIP prefers the background regions than the foregrounds according to the
predicted similarity map, which contradicts human understanding. Besides, there
are obvious noisy activations on the visualization results at irrelevant
positions. To address these two issues, we conduct in-depth analyses and reveal
the reasons with new findings and evidences. Based on these insights, we
propose the CLIP Surgery, a method that enables surgery-like modifications for
the inference architecture and features, for better explainability and
enhancement in multiple open-vocabulary tasks. The proposed method has
significantly improved the explainability of CLIP for both convolutional
networks and vision transformers, surpassing existing methods by large margins.
Besides, our approach also demonstrates remarkable improvements in
open-vocabulary segmentation and multi-label recognition tasks. For examples,
the mAP improvement on NUS-Wide multi-label recognition is 4.41% without any
additional training, and our CLIP Surgery surpasses the state-of-the-art method
by 8.74% at mIoU on Cityscapes open-vocabulary semantic segmentation.
Furthermore, our method benefits other tasks including multimodal visualization
and interactive segmentation like Segment Anything Model (SAM). The code is
available at this https URL

CLIP（Contrastive Language-Image Pre-training）是一种强大的多模态视觉模型，该论文提出了一种 CLIP Surgery 方法，可以在不降低性能的情况下提升 CLIP 的解释性和性能，并在开放词汇任务中获得了显着的提高，如 NUS-Wide 多标签识别上得到了 4.41% 的平均精度提升，Cityscapes 开放词汇语义分割任务上的 mIoU 也超过了现有方法的 8.74%。