Image retrieval with hybrid-modality queries, also known as composing text
and image for image retrieval (CTI-IR), is a retrieval task where the search
intention is expressed in a more complex query format, involving both vision
and text modalities. For example, a target product image is searched using a
reference product image along with text about changing certain attributes of
the reference image as the query. It is a more challenging image retrieval task
that requires both semantic space learning and cross-modal fusion. Previous
approaches that attempt to deal with both aspects achieve unsatisfactory
performance. In this paper, we decompose the CTI-IR task into a three-stage
learning problem to progressively learn the complex knowledge for image
retrieval with hybrid-modality queries. We first leverage the semantic
embedding space for open-domain image-text retrieval, and then transfer the
learned knowledge to the fashion-domain with fashion-related pre-training
tasks. Finally, we enhance the pre-trained model from single-query to
hybrid-modality query for the CTI-IR task. Furthermore, as the contribution of
individual modality in the hybrid-modality query varies for different retrieval
scenarios, we propose a self-supervised adaptive weighting strategy to
dynamically determine the importance of image and text in the hybrid-modality
query for better retrieval. Extensive experiments show that our proposed model
significantly outperforms state-of-the-art methods in the mean of Recall@K by
24.9% and 9.5% on the Fashion-IQ and Shoes benchmark datasets respectively.

本论文介绍一种将语音和图像相结合进行图像检索的复杂检索方法，从而需要语义空间学习和跨模态融合，同时提出了自我监督适应权重策略以更好地检索，通过广泛的实验，表明该方法在 Fashion-IQ 和 Shoes 基准数据集上的召回率均比现有方法提高了 24.9％和 9.5％。