Sketches are highly expressive, inherently capturing subjective and
fine-grained visual cues. The exploration of such innate properties of human
sketches has, however, been limited to that of image retrieval. In this paper,
for the first time, we cultivate the expressiveness of sketches but for the
fundamental vision task of object detection. The end result is a sketch-enabled
object detection framework that detects based on what \textit{you} sketch --
\textit{that} ``zebra'' (e.g., one that is eating the grass) in a herd of
zebras (instance-aware detection), and only the \textit{part} (e.g., ``head" of
a ``zebra") that you desire (part-aware detection). We further dictate that our
model works without (i) knowing which category to expect at testing (zero-shot)
and (ii) not requiring additional bounding boxes (as per fully supervised) and
class labels (as per weakly supervised). Instead of devising a model from the
ground up, we show an intuitive synergy between foundation models (e.g., CLIP)
and existing sketch models build for sketch-based image retrieval (SBIR), which
can already elegantly solve the task -- CLIP to provide model generalisation,
and SBIR to bridge the (sketch$\rightarrow$photo) gap. In particular, we first
perform independent prompting on both sketch and photo branches of an SBIR
model to build highly generalisable sketch and photo encoders on the back of
the generalisation ability of CLIP. We then devise a training paradigm to adapt
the learned encoders for object detection, such that the region embeddings of
detected boxes are aligned with the sketch and photo embeddings from SBIR.
Evaluating our framework on standard object detection datasets like PASCAL-VOC
and MS-COCO outperforms both supervised (SOD) and weakly-supervised object
detectors (WSOD) on zero-shot setups. Project Page:
https://pinakinathc.github.io/sketch-detect

本研究介绍一种基于草图的物体检测框架，利用基础模型（如 CLIP）和既有的草图模型（如 SBIR）之间的协作来构建高度可推广的草图和照片编码器，并设计一种训练范式来适应对于物体检测，评估结果显示该框架在零样本设置下的表现优于有监督的和弱监督的对象检测器。

人类素描对于物体检测有何作用？

What Can Human Sketches Do for Object Detection?

Sketch-based image retrieval (SBIR) is the task of retrieving natural images
(photos) that match the semantics and the spatial configuration of hand-drawn
sketch queries. The universality of sketches extends the scope of possible
applications and increases the demand for efficient SBIR solutions. In this
paper, we study classic triplet-based SBIR solutions and show that a persistent
invariance to horizontal flip (even after model finetuning) is harming
performance. To overcome this limitation, we propose several approaches and
evaluate in depth each of them to check their effectiveness. Our main
contributions are twofold: We propose and evaluate several intuitive
modifications to build SBIR solutions with better flip equivariance. We show
that vision transformers are more suited for the SBIR task, and that they
outperform CNNs with a large margin. We carried out numerous experiments and
introduce the first models to outperform human performance on a large-scale
SBIR benchmark (Sketchy). Our best model achieves a recall of 62.25% (at k = 1)
on the sketchy benchmark compared to previous state-of-the-art methods 46.2%.

本文介绍了以往三元组 SBIR 方案存在的问题，提出了建立具有更好翻转等变性的 SBIR 解决方案的多种方法，并深入评估了每种方法的有效性，揭示了视觉 Transformer 对于 SBIR 任务更加适用，性能比卷积神经网络高得多，引入了第一个在大规模 SBIR 基准（Sketchy）上超越人类表现的模型，具体表现为召回率 62.25％，比以前的最先进方法 46.2％高。

Transformer 和 CNN 都在 SBIR 上击败了人类

Transformers and CNNs both Beat Humans on SBIR

Sketch-based image retrieval (SBIR) is the task of retrieving images from a
natural image database that correspond to a given hand-drawn sketch. Ideally,
an SBIR model should learn to associate components in the sketch (say, feet,
tail, etc.) with the corresponding components in the image having similar shape
characteristics. However, current evaluation methods simply focus only on
coarse-grained evaluation where the focus is on retrieving images which belong
to the same class as the sketch but not necessarily having the same shape
characteristics as in the sketch. As a result, existing methods simply learn to
associate sketches with classes seen during training and hence fail to
generalize to unseen classes. In this paper, we propose a new benchmark for
zero-shot SBIR where the model is evaluated in novel classes that are not seen
during training. We show through extensive experiments that existing models for
SBIR that are trained in a discriminative setting learn only class specific
mappings and fail to generalize to the proposed zero-shot setting. To
circumvent this, we propose a generative approach for the SBIR task by
proposing deep conditional generative models that take the sketch as an input
and fill the missing information stochastically. Experiments on this new
benchmark created from the "Sketchy" dataset, which is a large-scale database
of sketch-photo pairs demonstrate that the performance of these generative
models is significantly better than several state-of-the-art approaches in the
proposed zero-shot framework of the coarse-grained SBIR task.

提出基于生成模型的零样本学习用于手绘图像检索的新方法，在经过大量实验验证后，发现这种方法比现有的方法在检索不同类别的图片时性能更好。