We investigate the reasoning ability of pretrained vision and language (V&L)
models in two tasks that require multimodal integration: (1) discriminating a
correct image-sentence pair from an incorrect one, and (2) counting entities in
an image. We evaluate three pretrained V&L models on these tasks: ViLBERT,
ViLBERT 12-in-1 and LXMERT, in zero-shot and finetuned settings. Our results
show that models solve task (1) very well, as expected, since all models are
pretrained on task (1). However, none of the pretrained V&L models is able to
adequately solve task (2), our counting probe, and they cannot generalise to
out-of-distribution quantities. We propose a number of explanations for these
findings: LXMERT (and to some extent ViLBERT 12-in-1) show some evidence of
catastrophic forgetting on task (1). Concerning our results on the counting
probe, we find evidence that all models are impacted by dataset bias, and also
fail to individuate entities in the visual input. While a selling point of
pretrained V&L models is their ability to solve complex tasks, our findings
suggest that understanding their reasoning and grounding capabilities requires
more targeted investigations on specific phenomena.

本研究探讨了预训练视觉和语言模型在两个需要多模态集成的任务中的推理能力：（1）区分正确的图像 - 句子对与不正确的对，以及（2）计算图像中的实体。结果显示，预训练的 V＆L 模型非常擅长解决任务（1），但无法充分解决任务（2），即计数探针，并不能推广到不同分布的数量。研究为我们提供了一些对这些发现的解释，并建议理解这些模型的推理和基础能力需要更有目的的调查。