Standard practice in pretraining multimodal models, such as vision-language
models, is to rely on pairs of aligned inputs from both modalities, for
example, aligned image-text pairs. However, such pairs can be difficult to
obtain in low-resource settings and for some modality pairs (e.g., structured
tables and images). In this work, we investigate the extent to which we can
reduce the reliance on such parallel data, which we term \emph{bimodal
supervision}, and use models that are pretrained on each modality
independently. We experiment with a high-performing vision-language model, and
analyze the effect of bimodal supervision on three vision-language tasks. We
find that on simpler tasks, such as VQAv2 and GQA, one can eliminate bimodal
supervision completely, suffering only a minor loss in performance. Conversely,
for NLVR2, which requires more complex reasoning, training without bimodal
supervision leads to random performance. Nevertheless, using only 5\% of the
bimodal data (142K images along with their captions), or leveraging weak
supervision in the form of a list of machine-generated labels for each image,
leads to only a moderate degradation compared to using 3M image-text pairs:
74\%$\rightarrow$$\sim$70\%. Our code is available at
this https URL.

本次研究旨在探究如何在预训练视觉 - 语言模型中减少对平行数据的依赖，通过对高性能视觉 - 语言模型的实验，发现在一些简单任务中完全可以消除双模态监督，但在更复杂的任务中，没有双模态监督将导致随机的性能。然而，利用 5％的双模态数据或弱监督产生的标签仅会引起中等程度的性能下降。