Weakly supervised text-based person re-identification (TPRe-ID) seeks to
retrieve images of a target person using textual descriptions, without relying
on identity annotations and is more challenging and practical. The primary
challenge is the intra-class differences, encompassing intra-modal feature
variations and cross-modal semantic gaps. Prior works have focused on
instance-level samples and ignored prototypical features of each person which
are intrinsic and invariant. Toward this, we propose a Cross-Modal Prototypical
Contrastive Learning (CPCL) method. In practice, the CPCL introduces the CLIP
model to weakly supervised TPRe-ID for the first time, mapping visual and
textual instances into a shared latent space. Subsequently, the proposed
Prototypical Multi-modal Memory (PMM) module captures associations between
heterogeneous modalities of image-text pairs belonging to the same person
through the Hybrid Cross-modal Matching (HCM) module in a many-to-many mapping
fashion. Moreover, the Outlier Pseudo Label Mining (OPLM) module further
distinguishes valuable outlier samples from each modality, enhancing the
creation of more reliable clusters by mining implicit relationships between
image-text pairs. Experimental results demonstrate that our proposed CPCL
attains state-of-the-art performance on all three public datasets, with a
significant improvement of 11.58%, 8.77% and 5.25% in Rank@1 accuracy on
CUHK-PEDES, ICFG-PEDES and RSTPReid datasets, respectively. The code is
available at this https URL

提出了一种基于弱监督的文本驱动的人物重识别方法，使用跨模态原型对比学习和多模态记忆模块来改善检索性能。

CPCL：跨模态原型对比学习弱监督的基于文本的人员再识别

CPCL: Cross-Modal Prototypical Contrastive Learning for Weakly  Supervised Text-based Person Re-Identification

Text-based person re-identification (TBPReID) aims to retrieve person images
represented by a given textual query. In this task, how to effectively align
images and texts globally and locally is a crucial challenge. Recent works have
obtained high performances by solving Masked Language Modeling (MLM) to align
image/text parts. However, they only performed uni-directional (i.e., from
image to text) local-matching, leaving room for improvement by introducing
opposite-directional (i.e., from text to image) local-matching. In this work,
we introduce Bidirectional Local-Matching (BiLMa) framework that jointly
optimize MLM and Masked Image Modeling (MIM) in TBPReID model training. With
this framework, our model is trained so as the labels of randomly masked both
image and text tokens are predicted by unmasked tokens. In addition, to narrow
the semantic gap between image and text in MIM, we propose Semantic MIM
(SemMIM), in which the labels of masked image tokens are automatically given by
a state-of-the-art human parser. Experimental results demonstrate that our
BiLMa framework with SemMIM achieves state-of-the-art Rank@1 and mAP scores on
three benchmarks.

通过引入双向本地匹配 (BiLMa) 框架，我们在文本输入的图像识别任务中提出了 Masked Language Modeling (MLM) 和 Masked Image Modeling (MIM) 的联合优化方法，同时引入 Semantic MIM (SemMIM) 来减小图像和文本之间的语义差距。实验证明，我们的 BiLMa 框架与 SemMIM 在三个评估基准上达到了最先进的 Rank@1 和 mAP 得分。