Vision-and-Language Pretraining (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches for VLP heavily rely on image feature extraction processes, most of which involve region supervisions (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the actual multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual encoder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to 60 times faster than previous VLP models, yet with competitive or better downstream task performance.

本文提出了一种新的Vision-and-Language Pre-training模型ViLT，它是一种单体模型，与文本输入处理方式相同，并通过多模态交互步骤实现视觉输入处理。ViLT通过简化图像输入处理过程，使得模型训练更加高效，可以有效地提高下游任务的性能表现。

ViLT：无卷积或区域监督的视觉语言Transformer