We introduce a new pre-trainable generic representation for visual-linguistic
tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the
simple yet powerful Transformer model as the backbone, and extends it to take
both visual and linguistic embedded features as input. In it, each element of
the input is either of a word from the input sentence, or a region-of-interest
(RoI) from the input image. It is designed to fit for most of the
visual-linguistic downstream tasks. To better exploit the generic
representation, we pre-train VL-BERT on the massive-scale Conceptual Captions
dataset, together with text-only corpus. Extensive empirical analysis
demonstrates that the pre-training procedure can better align the
visual-linguistic clues and benefit the downstream tasks, such as visual
commonsense reasoning, visual question answering and referring expression
comprehension. It is worth noting that VL-BERT achieved the first place of
single model on the leaderboard of the VCR benchmark. Code is released at
https://github.com/jackroos/VL-BERT.

本研究引入一个新的预可训练的通用视觉语言表示方法 ——Visual-Linguistic BERT，它采用了简单而强大的 Transformer 模型作为骨干网络，并将视觉和语言嵌入特征扩展为输入。通过在大规模的 Conceptual Captions 数据集上进行文本预训练，VL-BERT 可以适配大多数视觉语言下游任务，并在可视化常识推理、视觉问答、指称理解等下游任务中取得了不错的效果。