Large-scale pre-training methods of learning cross-modal representations on
image-text pairs are becoming popular for vision-language tasks. While existing
methods simply concatenate image region features and text features as input to
the model to be pre-trained and use self-attention to learn image-text semantic
alignments in a brute force manner, in this paper, we propose a new learning
method Oscar (Object-Semantics Aligned Pre-training), which uses object tags
detected in images as anchor points to significantly ease the learning of
alignments. Our method is motivated by the observation that the salient objects
in an image can be accurately detected, and are often mentioned in the paired
text. We pre-train an Oscar model on the public corpus of 6.5 million
text-image pairs, and fine-tune it on downstream tasks, creating new
state-of-the-arts on six well-established vision-language understanding and
generation tasks.

本文介绍了一种基于目标检测的图像 - 文本预训练学习方法 Oscar，通过使用在图像中检测到的目标标签作为锚点，显著简化模型中的图像与文本的语义对齐，使其在六个视觉 - 语言理解和生成任务中创造了新的最好成果。