This paper proposes a novel zero-shot composed image retrieval (CIR) method
considering the query-target relationship by masked image-text pairs. The
objective of CIR is to retrieve the target image using a query image and a
query text. Existing methods use a textual inversion network to convert the
query image into a pseudo word to compose the image and text and use a
pre-trained visual-language model to realize the retrieval. However, they do
not consider the query-target relationship to train the textual inversion
network to acquire information for retrieval. In this paper, we propose a novel
zero-shot CIR method that is trained end-to-end using masked image-text pairs.
By exploiting the abundant image-text pairs that are convenient to obtain with
a masking strategy for learning the query-target relationship, it is expected
that accurate zero-shot CIR using a retrieval-focused textual inversion network
can be realized. Experimental results show the effectiveness of the proposed
method.

提出了一种考虑查询目标关系的零样本组合图像检索方法，通过遮蔽的图像文本对。通过利用遮蔽策略学习查询目标关系并训练旨在检索的文本反转网络，预期可以实现精确的零样本组合图像检索。实验结果显示了该方法的有效性。

零样本组图检索，考虑到查询目标之间的关系，利用遮罩图像文本对

Zero-shot Composed Image Retrieval Considering Query-target Relationship  Leveraging Masked Image-text Pairs

Searching for specific person has great security value and social benefits,
and it often involves a combination of visual and textual information.
Conventional person retrieval methods, whether image-based or text-based,
usually fall short in effectively harnessing both types of information, leading
to the loss of accuracy. In this paper, a whole new task called Composed Person
Retrieval (CPR) is proposed to jointly utilize both image and text information
for target person retrieval. However, the supervised CPR must depend on very
costly manual annotation dataset, while there are currently no available
resources. To mitigate this issue, we firstly introduce the Zero-shot Composed
Person Retrieval (ZS-CPR), which leverages existing domain-related data to
resolve the CPR problem without reliance on expensive annotations. Secondly, to
learn ZS-CPR model, we propose a two-stage learning framework, Word4Per, where
a lightweight Textual Inversion Network (TINet) and a text-based person
retrieval model based on fine-tuned Contrastive Language-Image Pre-training
(CLIP) network are learned without utilizing any CPR data. Thirdly, a finely
annotated Image-Text Composed Person Retrieval dataset (ITCPR) is built as the
benchmark to assess the performance of the proposed Word4Per framework.
Extensive experiments under both Rank-1 and mAP demonstrate the effectiveness
of Word4Per for the ZS-CPR task, surpassing the comparative methods by over
10%. The code and ITCPR dataset will be publicly available at
this https URL

提出了一种新的任务，称为组合人员检索（CPR），通过整合图像和文本信息实现目标人员的检索；通过 Zero-shot Composed Person Retrieval（ZS-CPR）方法，利用现有领域相关数据解决了高代价手动注释数据资源的问题；通过 Word4Per 框架，学习了轻量级的文本逆转网络（TINet）和基于微调的对比语言 - 图像预训练网络（CLIP）的基于文本的人员检索模型，构建了一个精细注释的图像 - 文本组合人员检索数据集（ITCPR），并在 Rank-1 和 mAP 指标下的大量实验中证明了 Word4Per 在 ZS-CPR 任务上的有效性，超过比较方法 10% 以上的性能。