Most self-supervised video representation learning approaches focus on action
recognition. In contrast, in this paper we focus on self-supervised video
learning for movie understanding and propose a novel hierarchical
self-supervised pretraining strategy that separately pretrains each level of
our hierarchical movie understanding model (based on [37]). Specifically, we
propose to pretrain the low-level video backbone using a contrastive learning
objective, while pretrain the higher-level video contextualizer using an event
mask prediction task, which enables the usage of different data sources for
pretraining different levels of the hierarchy. We first show that our
self-supervised pretraining strategies are effective and lead to improved
performance on all tasks and metrics on VidSitu benchmark [37] (e.g., improving
on semantic role prediction from 47% to 61% CIDEr scores). We further
demonstrate the effectiveness of our contextualized event features on LVU tasks
[54], both when used alone and when combined with instance features, showing
their complementarity.

本文介绍了一种面向电影理解的自监督视频学习方法，采用分层的预训练策略，在低层进行对比学习，高层则采用事件遮罩预测任务来预训练视频上下文模型，并在 VidSitu 基准测试中表现出更好的性能。同时，在 LVU 任务中，我们还展示了上下文化事件特征的有效性。

电影理解的分层自监督表征学习

Hierarchical Self-supervised Representation Learning for Movie  Understanding

Recent years have seen remarkable advances in visual understanding. However,
how to understand a story-based long video with artistic styles, e.g. movie,
remains challenging. In this paper, we introduce MovieNet -- a holistic dataset
for movie understanding. MovieNet contains 1,100 movies with a large amount of
multi-modal data, e.g. trailers, photos, plot descriptions, etc. Besides,
different aspects of manual annotations are provided in MovieNet, including
1.1M characters with bounding boxes and identities, 42K scene boundaries, 2.5K
aligned description sentences, 65K tags of place and action, and 92K tags of
cinematic style. To the best of our knowledge, MovieNet is the largest dataset
with richest annotations for comprehensive movie understanding. Based on
MovieNet, we set up several benchmarks for movie understanding from different
angles. Extensive experiments are executed on these benchmarks to show the
immeasurable value of MovieNet and the gap of current approaches towards
comprehensive movie understanding. We believe that such a holistic dataset
would promote the researches on story-based long video understanding and
beyond. MovieNet will be published in compliance with regulations at
this https URL

本文介绍了 MovieNet，这是一个综合数据集，包括 1100 部电影，大量多模态数据以及详尽的手动注释，可以被用于全面理解电影。

MovieNet: 电影理解的全面数据集

MovieNet: A Holistic Dataset for Movie Understanding

Inspired by the remarkable advances in video analytics, research teams are
stepping towards a greater ambition -- movie understanding. However, compared
to those activity videos in conventional datasets, movies are significantly
different. Generally, movies are much longer and consist of much richer
temporal structures. More importantly, the interactions among characters play a
central role in expressing the underlying story. To facilitate the efforts
along this direction, we construct a dataset called Movie Synopses Associations
(MSA) over 327 movies, which provides a synopsis for each movie, together with
annotated associations between synopsis paragraphs and movie segments. On top
of this dataset, we develop a framework to perform matching between movie
segments and synopsis paragraphs. This framework integrates different aspects
of a movie, including event dynamics and character interactions, and allows
them to be matched with parsed paragraphs, based on a graph-based formulation.
Our study shows that the proposed framework remarkably improves the matching
accuracy over conventional feature-based methods. It also reveals the
importance of narrative structures and character interactions in movie
understanding.

通过构建 327 个电影的电影简介关联（Movie Synopses Associations，MSA）数据集，以及开发一套能够识别电影片段并且和简介段落进行精准匹配的框架，本研究指出利用文学结构和人物互动来提高匹配精度是非常重要的，相较于传统的特征匹配方法，这种基于图形的综合匹配策略能够在电影理解领域有实质性的提升。