Existing image retrieval systems use text queries to provide a natural and practical way for users to express what they are looking for. However, fine-grained image retrieval often requires the ability to also express the where in the image the content they are looking for is. The textual modality can only cumbersomely express such localization preferences, whereas pointing would be a natural fit. In this paper, we describe an image retrieval setup where the user simultaneously describes an image using both spoken natural language (the "what") and mouse traces over an empty canvas (the "where") to express the characteristics of the desired target image. To this end, we learn an image retrieval model using the Localized Narratives dataset, which is capable of performing early fusion between text descriptions and synchronized mouse traces. Qualitative and quantitative experiments show that our model is capable of taking this spatial guidance into account, and provides more accurate retrieval results compared to text-only equivalent systems.

本文提出了一种新的多模态查询图像检索系统，并将用户同时使用自然语言和鼠标轨迹来表达所需目标图像的特征，本系统在现有图像检索模型的基础上进行了简单的修改，以更好地考虑所提供的空间引导。实验结果表明，本模型能够有效地利用空间引导，相比于仅使用文本查询的系统，本模型能提供更准确的图像检索结果。

多模式查询图像检索时传递主题语义和指示区域信息