Vision-language (V+L) pretraining models have achieved great success in
supporting multimedia applications by understanding the alignments between
images and text. While existing vision-language pretraining models primarily
focus on understanding objects in images or entities in text, they often ignore
the alignment at the level of events and their argument structures. In this
work, we propose a contrastive learning framework to enforce vision-language
pretraining models to comprehend events and associated argument (participant)
roles. To achieve this, we take advantage of text information extraction
technologies to obtain event structural knowledge, and utilize multiple prompt
functions to contrast difficult negative descriptions by manipulating event
structures. We also design an event graph alignment loss based on optimal
transport to capture event argument structures. In addition, we collect a large
event-rich dataset (106,875 images) for pretraining, which provides a more
challenging image retrieval benchmark to assess the understanding of
complicated lengthy sentences. Experiments show that our zero-shot CLIP-Event
outperforms the state-of-the-art supervised model in argument extraction on
Multimedia Event Extraction, achieving more than 5% absolute F-score gain in
event extraction, as well as significant improvements on a variety of
downstream tasks under zero-shot settings.

本文提出了一种对视觉语言预训练模型进行对事件及相关参数角色理解的对比学习框架，并结合文本信息提取技术，利用多个提示函数来构造事件结构，设计基于最优传输的事件图形对齐损失来捕获事件参数结构，通过无监督学习进行预训练，并在零样本的情况下取得了优秀的性能。