Despite the impressive performance achieved by pre-trained
language-and-vision models in downstream tasks, it remains an open question
whether this reflects a proper understanding of image-text interaction. In this
work, we explore to what extent they handle basic linguistic constructions --
active-passive voice, coordination, and relative clauses -- that even preschool
children can typically master. We present BLA, a novel, automatically
constructed benchmark to evaluate multimodal models on these Basic Language
Abilities. We show that different types of Transformer-based systems, such as
CLIP, ViLBERT, and BLIP2, generally struggle with BLA in a zero-shot setting,
in line with previous findings. Our experiments, in particular, show that most
of the tested models only marginally benefit when fine-tuned or prompted with
construction-specific samples. Yet, the generative BLIP2 shows promising
trends, especially in an in-context learning setting. This opens the door to
using BLA not only as an evaluation benchmark but also to improve models' basic
language abilities.

我们探索了预训练语言与视觉模型在基本语言构造方面的处理程度，并提出了 BLA 评估基准，发现各种类型的基于 Transformer 的系统普遍在零样本设置下难以处理 BLA，但生成型的 BLIP2 在上下文学习环境中显示出有希望的趋势，这为将 BLA 用作评估基准以及提高模型的基本语言能力打开了大门。