We study the underexplored but fundamental vision problem of machine
understanding of abstract freehand scene sketches. We introduce a sketch
encoder that results in semantically-aware feature space, which we evaluate by
testing its performance on a semantic sketch segmentation task. To train our
model we rely only on the availability of bitmap sketches with their brief
captions and do not require any pixel-level annotations. To obtain
generalization to a large set of sketches and categories, we build on a vision
transformer encoder pretrained with the CLIP model. We freeze the text encoder
and perform visual-prompt tuning of the visual encoder branch while introducing
a set of critical modifications. Firstly, we augment the classical key-query
(k-q) self-attention blocks with value-value (v-v) self-attention blocks.
Central to our model is a two-level hierarchical network design that enables
efficient semantic disentanglement: The first level ensures holistic scene
sketch encoding, and the second level focuses on individual categories. We,
then, in the second level of the hierarchy, introduce a cross-attention between
textual and visual branches. Our method outperforms zero-shot CLIP pixel
accuracy of segmentation results by 37 points, reaching an accuracy of $85.5\%$
on the FS-COCO sketch dataset. Finally, we conduct a user study that allows us
to identify further improvements needed over our method to reconcile machine
and human understanding of scene sketches.

我们研究了机器对抽象手绘场景草图的理解这一未被充分探索但基本的视觉问题。我们介绍了一种草图编码器，其产生了一个语义感知的特征空间，并通过对语义草图分割任务的性能进行评估。为了训练我们的模型，我们仅依赖于具有简要标题的位图草图的可用性，并且不需要任何像素级的注释。为了实现对大量草图和类别的泛化，我们建立在预先训练的 CLIP 模型上的视觉变换编码器的基础上。我们冻结文本编码器，并通过引入一组关键的修改来执行视觉提示微调视觉编码器分支。我们提供了一个两级分层网络设计，实现了高效的语义解耦：第一级确保了整体场景草图编码，第二级专注于个别类别。然后，在层次结构的第二级中，我们引入了文本和视觉分支之间的交叉注意。我们的方法超越了无标注 CLIP 像素分割结果的 37 个点，达到 FS-COCO 草图数据集上 85.5％的准确率。最后，我们进行了用户研究，以确定我们的方法在调和机器和人类对场景草图的理解方面还需要进一步改进。