We identify a critical bias in contemporary CLIP-based models, which we
denote as \textit{single tag bias}. This bias manifests as a disproportionate
focus on a singular tag (word) while neglecting other pertinent tags, stemming
from CLIP's text embeddings that prioritize one specific tag in image-text
relationships. When deconstructing text into individual tags, only one tag
tends to have high relevancy with CLIP's image embedding, leading to an
imbalanced tag relevancy. This results in an uneven alignment among multiple
tags present in the text. To tackle this challenge, we introduce a novel
two-step fine-tuning approach. First, our method leverages the similarity
between tags and their nearest pixels for scoring, enabling the extraction of
image-relevant tags from the text. Second, we present a self-distillation
strategy aimed at aligning the combined masks from extracted tags with the
text-derived mask. This approach mitigates the single tag bias, thereby
significantly improving the alignment of CLIP's model without necessitating
additional data or supervision. Our technique demonstrates model-agnostic
improvements in multi-tag classification and segmentation tasks, surpassing
competing methods that rely on external resources. Code is available at
this https URL

我们提出了一种新颖的二步微调方法，通过利用标签与其最近的像素之间的相似性进行评分，从文本中提取与图像相关的标签，并通过自蒸馏策略来校准提取标签的组合掩码与文本派生掩码。这种方法缓解了单标签偏见，显著提高了 CLIP 模型的对齐性，并超过了依赖外部资源的竞争方法，在多标签分类和分割任务上展示了模型无关的改进。