Recent work in vision-and-language pretraining has investigated supervised
signals from object detection data to learn better, fine-grained multimodal
representations. In this work, we take a step further and explore how we add
supervision from small-scale visual relation data. In particular, we propose
two pretraining approaches to contextualise visual entities in a multimodal
setup. With verbalised scene graphs, we transform visual relation triplets into
structured captions, and treat them as additional views of images. With masked
relation prediction, we further encourage relating entities from visually
masked contexts. When applied to strong baselines pretrained on large amounts
of Web data, zero-shot evaluations on both coarse-grained and fine-grained
tasks show the efficacy of our methods in learning multimodal representations
from weakly-supervised relations data.

本文研究了使用小规模可视关系数据的预训练方法，包括使用场景图将可视关系三元组转换为结构化说明，以及使用掩模关系预测进一步鼓励从可视上下文中关联实体，并且证明了这些方法从弱监督关系数据中学习多模态表示的有效性。