Composed image retrieval (CIR) task takes a composed query of image and text,
aiming to search relative images for both conditions. Conventional CIR
approaches need a training dataset composed of triplets of query image, query
text, and target image, which is very expensive to collect. Several recent
works have worked on the zero-shot (ZS) CIR paradigm to tackle the issue
without using pre-collected triplets. However, the existing ZS-CIR methods show
limited backbone scalability and generalizability due to the lack of diversity
of the input texts during training. We propose a novel CIR framework, only
using language for its training. Our LinCIR (Language-only training for CIR)
can be trained only with text datasets by a novel self-supervision named
self-masking projection (SMP). We project the text latent embedding to the
token embedding space and construct a new text by replacing the keyword tokens
of the original text. Then, we let the new and original texts have the same
latent embedding vector. With this simple strategy, LinCIR is surprisingly
efficient and highly effective; LinCIR with CLIP ViT-G backbone is trained in
48 minutes and shows the best ZS-CIR performances on four different CIR
benchmarks, CIRCO, GeneCIS, FashionIQ, and CIRR, even outperforming supervised
method on FashionIQ. Code is available at this https URL

我们提出了一种只使用语言进行训练的新型 CIR 框架，名为 LinCIR，通过一种名为自掩蔽投影（SMP）的新颖自我监督方法，将文本的潜在嵌入投影到标记嵌入空间，并构建一个新的文本，替换原始文本中的关键词标记，从而使得新文本和原始文本具有相同的潜在嵌入向量。LinCIR 通过这种简单的策略，非常高效且高效，LinCIR 搭配 CLIP ViT-G 骨干网络在 48 分钟内进行训练，在 CIRCO、GeneCIS、FashionIQ 和 CIRR 四个不同的 CIR 基准上展现出最佳的零 - shot CIR 性能，甚至在 FashionIQ 上超过了有监督方法。