Zero-shot composed image retrieval (ZS-CIR), which aims to retrieve a target
image based on textual modifications to a reference image without triplet
labeling, has gained more and more attention. Current ZS-CIR research mainly
relies on two unlabeled pre-trained models: the vision-language model, e.g.,
CLIP, and the Pic2Word/textual inversion model. However, the pre-trained models
and CIR tasks have substantial discrepancies, where the pre-trained models
learn the similarities between vision and language but CIR aims to learn the
modifications of the image guided by text. In this paper, we introduce a novel
unlabeled and pre-trained masked tuning approach to reduce the gap between the
pre-trained model and the downstream CIR task. We first reformulate the
pre-trained vision-language contrastive learning as the CIR task, where we
randomly mask input image patches to generate $\langle$masked image, text,
image$\rangle$ triple from an image-text pair. Then, we propose a masked
tuning, which uses the text and the masked image to learn the modifications of
the original image. With such a simple design, it can learn to capture
fine-grained text-guided modifications. Extensive experimental results
demonstrate the significant superiority of our approach over the baseline
models on three ZS-CIR datasets, including FashionIQ, CIRR, and CIRCO.

本文介绍了一种新的非标注、预训练的掩码调整方法，以减少预训练模型与下游组合图像检索任务之间的差距，并通过将视觉 - 语言对比学习重新定义为组合图像检索任务，并提出掩码调整来学习原始图像的修改，进而捕捉到细粒度的文本引导修改。广泛的实验证明了我们方法在 FashionIQ、CIRR 和 CIRCO 三个组合图像检索数据集上相对于基线模型具有显著的优势。