Given a query consisting of a reference image and a relative caption,
Composed Image Retrieval (CIR) aims to retrieve target images visually similar
to the reference one while incorporating the changes specified in the relative
caption. The reliance of supervised methods on labor-intensive manually labeled
datasets hinders their broad applicability. In this work, we introduce a new
task, Zero-Shot CIR (ZS-CIR), that addresses CIR without the need for a labeled
training dataset. We propose an approach named iSEARLE (improved zero-Shot
composEd imAge Retrieval with textuaL invErsion) that involves mapping the
visual information of the reference image into a pseudo-word token in CLIP
token embedding space and combining it with the relative caption. To foster
research on ZS-CIR, we present an open-domain benchmarking dataset named CIRCO
(Composed Image Retrieval on Common Objects in context), the first CIR dataset
where each query is labeled with multiple ground truths and a semantic
categorization. The experimental results illustrate that iSEARLE obtains
state-of-the-art performance on three different CIR datasets -- FashionIQ,
CIRR, and the proposed CIRCO -- and two additional evaluation settings, namely
domain conversion and object composition. The dataset, the code, and the model
are publicly available at this https URL

基于图像和相关说明的复合图像检索（CIR）是旨在检索与参考图像在视觉上相似并包含相关说明中指定的变化的目标图像。本文引入了一项名为 Zero-Shot CIR（ZS-CIR）的新任务，该任务不需要有标签的训练数据集，提出了一种名为 iSEARLE（改进的无监督复合图像检索）的方法，该方法将参考图像的视觉信息映射到 CLIP 标记嵌入空间的伪词标记中，并与相关说明结合。为促进 ZS-CIR 的研究，我们提供了一个名为 CIRCO 的开放领域基准数据集，这是第一个每个查询都用多个真实值和语义分类进行标记的 CIR 数据集。实验结果表明，iSEARLE 在三个不同的 CIR 数据集（FashionIQ、CIRR 和 CIRCO）以及领域转换和对象组合的两种额外评估设置上都获得了最先进的性能。数据集、代码和模型在此 https URL 上公开可用。

iSEARLE：改进文本线性进行零样本组合图像检索

iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image  Retrieval

Composed Image Retrieval (CIR) aims to retrieve a target image based on a
query composed of a reference image and a relative caption that describes the
difference between the two images. The high effort and cost required for
labeling datasets for CIR hamper the widespread usage of existing methods, as
they rely on supervised learning. In this work, we propose a new task,
Zero-Shot CIR (ZS-CIR), that aims to address CIR without requiring a labeled
training dataset. Our approach, named zero-Shot composEd imAge Retrieval with
textuaL invErsion (SEARLE), maps the visual features of the reference image
into a pseudo-word token in CLIP token embedding space and integrates it with
the relative caption. To support research on ZS-CIR, we introduce an
open-domain benchmarking dataset named Composed Image Retrieval on Common
Objects in context (CIRCO), which is the first dataset for CIR containing
multiple ground truths for each query. The experiments show that SEARLE
exhibits better performance than the baselines on the two main datasets for CIR
tasks, FashionIQ and CIRR, and on the proposed CIRCO. The dataset, the code and
the model are publicly available at this https URL .

本文提出了一种不需要标记的数据集来解决 CIR 问题的新方法 Zero-Shot CIR，并介绍了一个基于视觉特征和文本描述的 SEARLE 方法，同时提出了一个名为 CIRCO 的真实场景数据集，实验表明 SEARLE 在 CIRCO 数据集上的表现优于基线方法。