Pre-trained vision-language models like CLIP have recently shown superior
performances on various downstream tasks, including image classification and
segmentation. However, in fine-grained image re-identification (ReID), the
labels are indexes, lacking concrete text descriptions. Therefore, it remains
to be determined how such models could be applied to these tasks. This paper
first finds out that simply fine-tuning the visual model initialized by the
image encoder in CLIP, has already obtained competitive performances in various
ReID tasks. Then we propose a two-stage strategy to facilitate a better visual
representation. The key idea is to fully exploit the cross-modal description
ability in CLIP through a set of learnable text tokens for each ID and give
them to the text encoder to form ambiguous descriptions. In the first training
stage, image and text encoders from CLIP keep fixed, and only the text tokens
are optimized from scratch by the contrastive loss computed within a batch. In
the second stage, the ID-specific text tokens and their encoder become static,
providing constraints for fine-tuning the image encoder. With the help of the
designed loss in the downstream task, the image encoder is able to represent
data as vectors in the feature embedding accurately. The effectiveness of the
proposed strategy is validated on several datasets for the person or vehicle
ReID tasks. Code is available at this https URL

本文提出了一种利用 CLIP 模型的文本 - 图像交互能力来解决细粒度图像重识别问题的方法，通过对学习的文本编码器给出模糊的文本描述来增强视觉表示，并通过一系列基于对比度损失的优化训练来优化文本令牌。