With recent progress in large-scale vision and language representation
learning, Vision Language Pretraining (VLP) models have achieved promising
improvements on various multi-modal downstream tasks. Albeit powerful, these
pre-training models still do not take advantage of world knowledge, which is
implicit in multi-modal data but comprises abundant and complementary
information. In this work, we propose a REtrieval-based knowledge Augmented
Vision Language Pre-training model (REAVL), which retrieves world knowledge
from knowledge graphs (KGs) and incorporates them in vision-language
pre-training. REAVL has two core components: a knowledge retriever that
retrieves knowledge given multi-modal data, and a knowledge-augmented model
that fuses multi-modal data and knowledge. By novelly unifying four
knowledge-aware self-supervised tasks, REAVL promotes the mutual integration of
multi-modal data and knowledge by fusing explicit knowledge with
vision-language pairs for masked multi-modal data modeling and KG relational
reasoning. Empirical experiments show that REAVL achieves new state-of-the-art
performance uniformly on knowledge-based vision-language understanding and
multimodal entity linking tasks, and competitive results on general
vision-language tasks while only using 0.2% pre-training data of the best
models.

该篇论文提出了一种基于知识图谱的视觉语言预训练模型 (REAVL)，能够检索多模态数据中的世界知识，并结合多模态数据和知识来完成预训练，以提高视觉语言理解和多模态实体链接任务的性能。