The dominant probing approaches rely on the zero-shot performance of
image-text matching tasks to gain a finer-grained understanding of the
representations learned by recent multimodal image-language transformer models.
The evaluation is carried out on carefully curated datasets focusing on
counting, relations, attributes, and others. This work introduces an
alternative probing strategy called guided masking. The proposed approach
ablates different modalities using masking and assesses the model's ability to
predict the masked word with high accuracy. We focus on studying multimodal
models that consider regions of interest (ROI) features obtained by object
detectors as input tokens. We probe the understanding of verbs using guided
masking on ViLBERT, LXMERT, UNITER, and VisualBERT and show that these models
can predict the correct verb with high accuracy. This contrasts with previous
conclusions drawn from image-text matching probing techniques that frequently
fail in situations requiring verb understanding. The code for all experiments
will be publicly available this https URL

本研究提出了一种指导掩蔽的探测方法，评估最近的多模态图像语言变形器模型的学习表示能力，重点研究考虑感兴趣区域（ROI）特征作为输入标记的多模态模型，通过指导掩蔽分析动词的理解能力，在 ViLBERT、LXMERT、UNITER 和 VisualBERT 模型中，我们展示出这些模型能够以高准确度预测正确的动词。

超越图像 - 文本匹配：多模态变换器中使用引导掩模进行动词理解

Beyond Image-Text Matching: Verb Understanding in Multimodal  Transformers Using Guided Masking

Understanding verbs is crucial to modelling how people and objects interact
with each other and the environment through space and time. Recently,
state-of-the-art video-language models based on CLIP have been shown to have
limited verb understanding and to rely extensively on nouns, restricting their
performance in real-world video applications that require action and temporal
understanding. In this work, we improve verb understanding for CLIP-based
video-language models by proposing a new Verb-Focused Contrastive (VFC)
framework. This consists of two main components: (1) leveraging pretrained
large language models (LLMs) to create hard negatives for cross-modal
contrastive learning, together with a calibration strategy to balance the
occurrence of concepts in positive and negative pairs; and (2) enforcing a
fine-grained, verb phrase alignment loss. Our method achieves state-of-the-art
results for zero-shot performance on three downstream tasks that focus on verb
understanding: video-text matching, video question-answering and video
classification. To the best of our knowledge, this is the first work which
proposes a method to alleviate the verb understanding problem, and does not
simply highlight it.

本研究提出了一个新的以动词为中心对比学习（Verb-Focused Contrastive，VFC）框架，以改善基于 CLIP 的视频语言模型的动词理解。该方法采用预训练的大型语言模型（LLMs）创建难样本进行跨模态对比学习，以及实施细粒度的动词短语对齐损失。该方法在三个下游任务上实现了零射击性能的最新成果，包括视频文本匹配、视频问答和视频分类。

动词在行动：提高视频语言模型中动词理解能力

Verbs in Action: Improving verb understanding in video-language models

Multimodal image-language transformers have achieved impressive results on a
variety of tasks that rely on fine-tuning (e.g., visual question answering and
image retrieval). We are interested in shedding light on the quality of their
pretrained representations -- in particular, if these models can distinguish
different types of verbs or if they rely solely on nouns in a given sentence.
To do so, we collect a dataset of image-sentence pairs (in English) consisting
of 421 verbs that are either visual or commonly found in the pretraining data
(i.e., the Conceptual Captions dataset). We use this dataset to evaluate
pretrained image-language transformers and find that they fail more in
situations that require verb understanding compared to other parts of speech.
We also investigate what category of verbs are particularly challenging.

本文研究多模态图像语言变换器的预训练表示质量，研究表明在需要谓语理解的情况下这些模型的表现不佳，通过图像 - 语句对数据集评估模型性能，分类词汇类型并找到特别具有挑战性的词汇类型。