A fundamental challenge faced by existing Fine-Grained Sketch-Based Image
Retrieval (FG-SBIR) models is the data scarcity -- model performances are
largely bottlenecked by the lack of sketch-photo pairs. Whilst the number of
photos can be easily scaled, each corresponding sketch still needs to be
individually produced. In this paper, we aim to mitigate such an upper-bound on
sketch data, and study whether unlabelled photos alone (of which they are many)
can be cultivated for performances gain. In particular, we introduce a novel
semi-supervised framework for cross-modal retrieval that can additionally
leverage large-scale unlabelled photos to account for data scarcity. At the
centre of our semi-supervision design is a sequential photo-to-sketch
generation model that aims to generate paired sketches for unlabelled photos.
Importantly, we further introduce a discriminator guided mechanism to guide
against unfaithful generation, together with a distillation loss based
regularizer to provide tolerance against noisy training samples. Last but not
least, we treat generation and retrieval as two conjugate problems, where a
joint learning procedure is devised for each module to mutually benefit from
each other. Extensive experiments show that our semi-supervised model yields
significant performance boost over the state-of-the-art supervised
alternatives, as well as existing methods that can exploit unlabelled photos
for FG-SBIR.

本论文研究了基于细粒度手绘图像检索模型中存在的数据稀缺问题，并介绍了一种新型的半监督框架，可以利用大规模未标记的照片来提升模型表现，其中心设计是一个顺序照片到草图生成模型，并引入了鉴别器和蒸馏损失来提高数据的质量。实验证明，新的半监督模型相比现有的监督模型和具有利用未标记的照片的现有方法产生了显著的性能提升。