Text-based person search (TBPS) aims to retrieve images of a specific person
from a large image gallery based on a natural language description. Existing
methods rely on massive annotated image-text data to achieve satisfactory
performance in fully-supervised learning. It poses a significant challenge in
practice, as acquiring person images from surveillance videos is relatively
easy, while obtaining annotated texts is challenging. The paper undertakes a
pioneering initiative to explore TBPS under the semi-supervised setting, where
only a limited number of person images are annotated with textual descriptions
while the majority of images lack annotations. We present a two-stage basic
solution based on generation-then-retrieval for semi-supervised TBPS. The
generation stage enriches annotated data by applying an image captioning model
to generate pseudo-texts for unannotated images. Later, the retrieval stage
performs fully-supervised retrieval learning using the augmented data.
Significantly, considering the noise interference of the pseudo-texts on
retrieval learning, we propose a noise-robust retrieval framework that enhances
the ability of the retrieval model to handle noisy data. The framework
integrates two key strategies: Hybrid Patch-Channel Masking (PC-Mask) to refine
the model architecture, and Noise-Guided Progressive Training (NP-Train) to
enhance the training process. PC-Mask performs masking on the input data at
both the patch-level and the channel-level to prevent overfitting noisy
supervision. NP-Train introduces a progressive training schedule based on the
noise level of pseudo-texts to facilitate noise-robust learning. Extensive
experiments on multiple TBPS benchmarks show that the proposed framework
achieves promising performance under the semi-supervised setting.

基于图像标注数据有限的半监督框架下，该论文提出了一个两阶段的基本解决方案，其中生成阶段利用图像字幕模型为未标注图片生成伪文本，检索阶段则使用增广数据进行完全监督的检索学习。而引入了噪声干扰的伪文本的框架，则提出了一个增强检索模型处理噪声数据能力的噪声鲁棒检索框架。该框架结合了混合 Patch-Channel 掩蔽和噪声引导渐进式训练两个关键策略，以防止过度拟合噪声监督，并根据伪文本的噪声水平引入渐进式训练进程。在多个 TBPS 基准测试上的广泛实验表明，该框架在半监督设置下取得了可观的性能。

半监督文本式人物搜索

Semi-supervised Text-based Person Search

Text-based person search (TBPS) aims to retrieve the images of the target
person from a large image gallery based on a given natural language
description. Existing methods are dominated by training models with parallel
image-text pairs, which are very costly to collect. In this paper, we make the
first attempt to explore TBPS without parallel image-text data ($\mu$-TBPS), in
which only non-parallel images and texts, or even image-only data, can be
adopted. Towards this end, we propose a two-stage framework,
generation-then-retrieval (GTR), to first generate the corresponding pseudo
text for each image and then perform the retrieval in a supervised manner. In
the generation stage, we propose a fine-grained image captioning strategy to
obtain an enriched description of the person image, which firstly utilizes a
set of instruction prompts to activate the off-the-shelf pretrained
vision-language model to capture and generate fine-grained person attributes,
and then converts the extracted attributes into a textual description via the
finetuned large language model or the hand-crafted template. In the retrieval
stage, considering the noise interference of the generated texts for training
model, we develop a confidence score-based training scheme by enabling more
reliable texts to contribute more during the training. Experimental results on
multiple TBPS benchmarks (i.e., CUHK-PEDES, ICFG-PEDES and RSTPReid) show that
the proposed GTR can achieve a promising performance without relying on
parallel image-text data.

本文提出了一种基于生成 - 检索（GTR）框架的文本搜索人物图像方法（TBPS），其中通过细粒度图像字幕和噪声评分训练来生成人物图像的伪文本，实现了不需要并行图像 - 文本数据的高效搜索。实验表明，所提出的方法在多个 TBPS 基准（即 CUHK-PEDES、ICFG-PEDES 和 RSTPReid）上具有很好的性能。