In this work we introduce a cross modal image retrieval system that allows
both text and sketch as input modalities for the query. A cross-modal deep
network architecture is formulated to jointly model the sketch and text input
modalities as well as the the image output modality, learning a common
embedding between text and images and between sketches and images. In addition,
an attention model is used to selectively focus the attention on the different
objects of the image, allowing for retrieval with multiple objects in the
query. Experiments show that the proposed method performs the best in both
single and multiple object image retrieval in standard datasets.

本文介绍了一种跨模态图像检索系统，允许文本和草图作为查询的输入模态。使用跨模态深度网络架构共同建模草图和文本输入模态以及图像输出模态，通过学习文本和图像之间以及草图和图像之间的共同嵌入，使用注意力模型有选择性地关注图像的不同对象，可以实现多对象检索。实验结果表明，所提出的方法在标准数据集中实现了最佳的单对象和多对象图像检索