Composed Image Retrieval (CoIR) has recently gained popularity as a task that
considers both text and image queries together, to search for relevant images
in a database. Most CoIR approaches require manually annotated datasets,
comprising image-text-image triplets, where the text describes a modification
from the query image to the target image. However, manual curation of CoIR
triplets is expensive and prevents scalability. In this work, we instead
propose a scalable automatic dataset creation methodology that generates
triplets given video-caption pairs, while also expanding the scope of the task
to include composed video retrieval (CoVR). To this end, we mine paired videos
with a similar caption from a large database, and leverage a large language
model to generate the corresponding modification text. Applying this
methodology to the extensive WebVid2M collection, we automatically construct
our WebVid-CoVR dataset, resulting in 1.6 million triplets. Moreover, we
introduce a new benchmark for CoVR with a manually annotated evaluation set,
along with baseline results. Our experiments further demonstrate that training
a CoVR model on our dataset effectively transfers to CoIR, leading to improved
state-of-the-art performance in the zero-shot setup on both the CIRR and
FashionIQ benchmarks. Our code, datasets, and models are publicly available at
this https URL

通过自动数据集创建方法，我们提出了一个可扩展的 Composed Image Retrieval（CoIR）任务，使用由视频标题对生成的三元组，扩展任务范围到 Composed Video Retrieval（CoVR）。通过在庞大的数据库中挖掘具有相似标题的配对视频，并利用大型语言模型生成相应的修改文本，我们构建了 WebVid-CoVR 数据集，其中包含 1.6 百万个三元组。实验证明，在我们的数据集上训练 CoVR 模型可以有效迁移到 CoIR，提高了 CIRR 和 FashionIQ 基准测试中的最新性能。

CoVR：从网络视频字幕中学习组合视频检索

CoVR: Learning Composed Video Retrieval from Web Video Captions

While interest in models that generalize at test time to new compositions has
risen in recent years, benchmarks in the visually-grounded domain have thus far
been restricted to synthetic images. In this work, we propose COVR, a new
test-bed for visually-grounded compositional generalization with real images.
To create COVR, we use real images annotated with scene graphs, and propose an
almost fully automatic procedure for generating question-answer pairs along
with a set of context images. COVR focuses on questions that require complex
reasoning, including higher-order operations such as quantification and
aggregation. Due to the automatic generation process, COVR facilitates the
creation of compositional splits, where models at test time need to generalize
to new concepts and compositions in a zero- or few-shot setting. We construct
compositional splits using COVR and demonstrate a myriad of cases where
state-of-the-art pre-trained language-and-vision models struggle to
compositionally generalize.

本文提出了 COVR，一个用于基于真实图像的组合通用化的新测试平台并使用基于场景图注释的实际图像和提出几乎完全自动的生成问题答案对以及相关 (场景) 图像集的过程来创建。由于自动生成过程，COVR 便于组合划分的生成，模型可以在测试时间零或少量训练数据的情况下进行新概念和组合的通用化。我们使用 COVR 构建了组合划分，并演示了许多情况，其中最先进的预训练语言和视觉模型难以进行组合通用化。