Large-scale pretraining and task-specific fine-tuning is now the standard methodology for many tasks in computer vision and natural language processing. Recently, a multitude of methods have been proposed for pretraining vision and language BERTs to tackle challenges at the intersection of these two key areas of AI. These models can be categorized into either single-stream or dual-stream encoders. We study the differences between these two categories, and show how they can be unified under a single theoretical framework. We then conduct controlled experiments to discern the empirical differences between five V&L BERTs. Our experiments show that training data and hyperparameters are responsible for most of the differences between the reported results, but they also reveal that the embedding layer plays a crucial role in these massive models.

本文探讨了计算机视觉和自然语言处理中预训练方法的标准方法学习经验，分别采用单流和双流编码器进行归纳分析。我们就五种视觉与语言BERT的差异进行经验研究，揭示出预训练数据和超参数是这些异常壮观的模型之间不同的原因，同时将两类网络统一在同一理论框架之下，指出embedding layer在这些模型中具有至关重要的作用。

多模态预训练揭示：视觉和语言 BERT 的元分析和统一框架