Learning from image-text data has demonstrated recent success for many
recognition tasks, yet is currently limited to visual features or individual
visual concepts such as objects. In this paper, we propose one of the first
methods that learn from image-sentence pairs to extract a graphical
representation of localized objects and their relationships within an image,
known as scene graph. To bridge the gap between images and texts, we leverage
an off-the-shelf object detector to identify and localize object instances,
match labels of detected regions to concepts parsed from captions, and thus
create "pseudo" labels for learning scene graph. Further, we design a
Transformer-based model to predict these "pseudo" labels via a masked token
prediction task. Learning from only image-sentence pairs, our model achieves
30% relative gain over a latest method trained with human-annotated unlocalized
scene graphs. Our model also shows strong results for weakly and fully
supervised scene graph generation. In addition, we explore an open-vocabulary
setting for detecting scene graphs, and present the first result for open-set
scene graph generation. Our code is available at
this https URL

本文提出了一种通过图像 - 句子对学习的方法，以提取图像中本地化对象及其关系的图形表示，即场景图，其中利用了现成的对象检测器来识别和定位对象实例，并通过掩码标记预测任务预测伪标签以学习场景图。通过学习图像 - 句子对，我们的模型相较于使用人工注释的非定位场景图训练的最新方法获得了 30％的相对增益。同时，在弱监督和全监督场景图生成方面，我们的模型也展现了出色的结果，我们还探究了用于检测场景图的开放词汇设置，并呈现了首个开放式场景图生成的结果。

从自然语言监督中学习生成场景图

Learning to Generate Scene Graph from Natural Language Supervision

Multimodal image-language transformers have achieved impressive results on a
variety of tasks that rely on fine-tuning (e.g., visual question answering and
image retrieval). We are interested in shedding light on the quality of their
pretrained representations -- in particular, if these models can distinguish
different types of verbs or if they rely solely on nouns in a given sentence.
To do so, we collect a dataset of image-sentence pairs (in English) consisting
of 421 verbs that are either visual or commonly found in the pretraining data
(i.e., the Conceptual Captions dataset). We use this dataset to evaluate
pretrained image-language transformers and find that they fail more in
situations that require verb understanding compared to other parts of speech.
We also investigate what category of verbs are particularly challenging.

本文研究多模态图像语言变换器的预训练表示质量，研究表明在需要谓语理解的情况下这些模型的表现不佳，通过图像 - 语句对数据集评估模型性能，分类词汇类型并找到特别具有挑战性的词汇类型。