Latent text representations exhibit geometric regularities, such as the
famous analogy: queen is to king what woman is to man. Such structured semantic
relations were not demonstrated on image representations. Recent works aiming
at bridging this semantic gap embed images and text into a multimodal space,
enabling the transfer of text-defined transformations to the image modality. We
introduce the SIMAT dataset to evaluate the task of Image Retrieval with
Multimodal queries. SIMAT contains 6k images and 18k textual transformation
queries that aim at either replacing scene elements or changing pairwise
relationships between scene elements. The goal is to retrieve an image
consistent with the (source image, text transformation) query. We use an
image/text matching oracle (OSCAR) to assess whether the image transformation
is successful. The SIMAT dataset will be publicly available. We use SIMAT to
evaluate the geometric properties of multimodal embedding spaces trained with
an image/text matching objective, like CLIP. We show that vanilla CLIP
embeddings are not very well suited to transform images with delta vectors, but
that a simple finetuning on the COCO dataset can bring dramatic improvements.
We also study whether it is beneficial to leverage pretrained universal
sentence encoders (FastText, LASER and LaBSE).

本文提供了一种检测基于多模态查询的图像检索任务的新数据集 SIMAT，使用图像 / 文本匹配预测器来评估基于图像 / 文本匹配策略训练的多模态嵌入空间的几何属性，并研究了利用预训练通用句子编码器（FastText，LASER 和 LaBSE）是否有益。