Computer vision tasks such as object detection and semantic/instance
segmentation rely on the painstaking annotation of large training datasets. In
this paper, we propose LocTex that takes advantage of the low-cost localized
textual annotations (i.e., captions and synchronized mouse-over gestures) to
reduce the annotation effort. We introduce a contrastive pre-training framework
between images and captions and propose to supervise the cross-modal attention
map with rendered mouse traces to provide coarse localization signals. Our
learned visual features capture rich semantics (from free-form captions) and
accurate localization (from mouse traces), which are very effective when
transferred to various downstream vision tasks. Compared with ImageNet
supervised pre-training, LocTex can reduce the size of the pre-training dataset
by 10x or the target dataset by 2x while achieving comparable or even improved
performance on COCO instance segmentation. When provided with the same amount
of annotations, LocTex achieves around 4% higher accuracy than the previous
state-of-the-art "vision+language" pre-training approach on the task of PASCAL
VOC image classification.

本文提出了一种名为 LocTex 的计算机视觉方法，它利用了低成本的本地化文本标注和鼠标轨迹，通过对图像和标题的对比预训练和监督跨模态关注图，从而提供粗略的本地化信号，从而减少标注数据的数量。它学习到的视觉特征可以捕捉自由形式的标题的丰富语义和鼠标轨迹的准确本地化，可以转移到各种下游视觉任务中，并且比 ImageNet 的监督预训练方法可以将预训练数据集的大小缩小 10 倍或目标数据集的大小缩小 2 倍，同时在 COCO 实例分割上实现可比较甚至更高的性能。在拥有相同数量的注释的情况下，LocTex 在 PASCAL VOC 图像分类任务上比以前最先进的 “视觉 + 语言” 预训练方法提高了约 4% 的准确性。