Recent advances in vision-and-language modeling have seen the development of
Transformer architectures that achieve remarkable performance on multimodal
reasoning tasks. Yet, the exact capabilities of these black-box models are
still poorly understood. While much of previous work has focused on studying
their ability to learn meaning at the word-level, their ability to track
syntactic dependencies between words has received less attention. We take a
first step in closing this gap by creating a new multimodal task targeted at
evaluating understanding of predicate-noun dependencies in a controlled setup.
We evaluate a range of state-of-the-art models and find that their performance
on the task varies considerably, with some models performing relatively well
and others at chance level. In an effort to explain this variability, our
analyses indicate that the quality (and not only sheer quantity) of pretraining
data is essential. Additionally, the best performing models leverage
fine-grained multimodal pretraining objectives in addition to the standard
image-text matching objectives. This study highlights that targeted and
controlled evaluations are a crucial step for a precise and rigorous test of
the multimodal knowledge of vision-and-language models.

本文研究视觉 - 语言建模，通过创建新的多模态任务和分析预训练数据的质量，发现预训练数据的质量和多模态预训练目标对模型的性能影响重要。