In this paper, we study the problem of zero-shot sketch-based image retrieval
(ZS-SBIR). The prior methods tackle the problem in a two-modality setting with
only category labels or even no textual information involved. However, the
growing prevalence of Large-scale pre-trained Language Models (LLMs), which
have demonstrated great knowledge learned from web-scale data, can provide us
with an opportunity to conclude collective textual information. Our key
innovation lies in the usage of text data as auxiliary information for images,
thus leveraging the inherent zero-shot generalization ability that language
offers. To this end, we propose an approach called Cross-Modal Attention
Alignment Network with Auxiliary Text Description for zero-shot sketch-based
image retrieval. The network consists of three components: (i) a Description
Generation Module that generates textual descriptions for each training
category by prompting an LLM with several interrogative sentences, (ii) a
Feature Extraction Module that includes two ViTs for sketch and image data, a
transformer for extracting tokens of sentences of each training category,
finally (iii) a Cross-modal Alignment Module that exchanges the token features
of both text-sketch and text-image using cross-attention mechanism, and align
the tokens locally and globally. Extensive experiments on three benchmark
datasets show our superior performances over the state-of-the-art ZS-SBIR
methods.

本文研究零样本基于手绘图像检索（ZS-SBIR）问题，利用大规模预训练语言模型为图像提供辅助文本描述，通过交叉注意力机制在文本 - 手绘图像和文本 - 真实图像之间进行特征交换和对齐，实现零样本泛化能力，实验结果显示在三个基准数据集上，相比现有的 ZS-SBIR 方法，我们具有卓越的性能。