Understanding how events described or shown in multimedia content relate to
one another is a critical component to developing robust artificially
intelligent systems which can reason about real-world media. While much
research has been devoted to event understanding in the text, image, and video
domains, none have explored the complex relations that events experience across
domains. For example, a news article may describe a `protest' event while a
video shows an `arrest' event. Recognizing that the visual `arrest' event is a
subevent of the broader `protest' event is a challenging, yet important problem
that prior work has not explored. In this paper, we propose the novel task of
MultiModal Event Event Relations to recognize such cross-modal event relations.
We contribute a large-scale dataset consisting of 100k video-news article
pairs, as well as a benchmark of densely annotated data. We also propose a
weakly supervised multimodal method which integrates commonsense knowledge from
an external knowledge base (KB) to predict rich multimodal event hierarchies.
Experiments show that our model outperforms a number of competitive baselines
on our proposed benchmark. We also perform a detailed analysis of our model's
performance and suggest directions for future research.

本研究提出了多模态事件关系的新任务，并开发了一个大规模数据集和一种基于外部知识库的弱监督多模态方法，为人工智能系统实现媒体理解和跨媒体事件关系建立提供了支持。