Several multi-modality representation learning approaches such as LXMERT and
ViLBERT have been proposed recently. Such approaches can achieve superior
performance due to the high-level semantic information captured during
large-scale multimodal pretraining. However, as ViLBERT and LXMERT adopt visual
region regression and classification loss, they often suffer from domain gap
and noisy label problems, based on the visual features having been pretrained
on the Visual Genome dataset. To overcome these issues, we propose unbiased
Contrastive Visual-Linguistic Pretraining (CVLP), which constructs a visual
self-supervised loss built upon contrastive learning. We evaluate CVLP on
several down-stream tasks, including VQA, GQA and NLVR2 to validate the
superiority of contrastive learning on multi-modality representation learning.
Our code is available at: this https URL

本文提出了一种基于对比学习的无偏置视觉 - 语言预训练方法，可以在多模态表示学习中获得更好的性能，在验证集 VQA、GQA 和 NLVR2 中取得了良好的结果。

对比视觉语言预训练

Contrastive Visual-Linguistic Pretraining

NLVR2 (Suhr et al., 2019) was designed to be robust for language bias through
a data collection process that resulted in each natural language sentence
appearing with both true and false labels. The process did not provide a
similar measure of control for visual bias. This technical report analyzes the
potential for visual bias in NLVR2. We show that some amount of visual bias
likely exists. Finally, we identify a subset of the test data that allows to
test for model performance in a way that is robust to such potential biases. We
show that the performance of existing models (Li et al., 2019; Tan and Bansal
2019) is relatively robust to this potential bias. We propose to add the
evaluation on this subset of the data to the NLVR2 evaluation protocol, and
update the official release to include it. A notebook including an
implementation of the code used to replicate this analysis is available at
this http URL

本技术报告分析了 NLVR2 的潜在视觉偏见，并提出通过测试可消除偏差来测试模型表现的数据子集。结果表明现有模型的表现相对稳健，因此建议将该子集数据纳入 NLVR2 的评估协议更新。