Sketch-based image retrieval (SBIR) is the task of retrieving natural images
(photos) that match the semantics and the spatial configuration of hand-drawn
sketch queries. The universality of sketches extends the scope of possible
applications and increases the demand for efficient SBIR solutions. In this
paper, we study classic triplet-based SBIR solutions and show that a persistent
invariance to horizontal flip (even after model finetuning) is harming
performance. To overcome this limitation, we propose several approaches and
evaluate in depth each of them to check their effectiveness. Our main
contributions are twofold: We propose and evaluate several intuitive
modifications to build SBIR solutions with better flip equivariance. We show
that vision transformers are more suited for the SBIR task, and that they
outperform CNNs with a large margin. We carried out numerous experiments and
introduce the first models to outperform human performance on a large-scale
SBIR benchmark (Sketchy). Our best model achieves a recall of 62.25% (at k = 1)
on the sketchy benchmark compared to previous state-of-the-art methods 46.2%.

本文介绍了以往三元组 SBIR 方案存在的问题，提出了建立具有更好翻转等变性的 SBIR 解决方案的多种方法，并深入评估了每种方法的有效性，揭示了视觉 Transformer 对于 SBIR 任务更加适用，性能比卷积神经网络高得多，引入了第一个在大规模 SBIR 基准（Sketchy）上超越人类表现的模型，具体表现为召回率 62.25％，比以前的最先进方法 46.2％高。