Recently multimodal transformer models have gained popularity because their
performance on language and vision tasks suggest they learn rich
visual-linguistic representations. Focusing on zero-shot image retrieval tasks,
we study three important factors which can impact the quality of learned
representations: pretraining data, the attention mechanism, and loss functions.
By pretraining models on six datasets, we observe that dataset noise and
language similarity to our downstream task are important indicators of model
performance. Through architectural analysis, we learn that models with a
multimodal attention mechanism can outperform deeper models with modality
specific attention mechanisms. Finally, we show that successful contrastive
losses used in the self-supervised learning literature do not yield similar
performance gains when used in multimodal transformers

本文章论述通过训练多模态 transformer 模型，其在语言和视觉任务上的表现证明了其可以学习到丰富的视觉 - 语言表达。其着重于零样本图像检索任务，并研究了三个重要因素：预训练数据、注意机制和损失函数，以评估其对于模型性能的影响。