Story video-text alignment, a core task in computational story understanding,
aims to align video clips with corresponding sentences in their descriptions.
However, progress on the task has been held back by the scarcity of manually
annotated video-text correspondence and the heavy concentration on English
narrations of Hollywood movies. To address these issues, in this paper, we
construct a large-scale multilingual video story dataset named Multilingual
Synopses of Movie Narratives (M-SYMON), containing 13,166 movie summary videos
from 7 languages, as well as manual annotation of fine-grained video-text
correspondences for 101.5 hours of video. Training on the human annotated data
from SyMoN outperforms the SOTA methods by 15.7 and 16.2 percentage points on
Clip Accuracy and Sentence IoU scores, respectively, demonstrating the
effectiveness of the annotations. As benchmarks for future research, we create
6 baseline approaches with different multilingual training strategies, compare
their performance in both intra-lingual and cross-lingual setups, exemplifying
the challenges of multilingual video-text alignment.

构建大规模多语言视频故事数据集 M-SYMON，包含 13166 个影片摘要视频和 101.5 小时视频的精细视频文本对应注释。使用 SyMoN 人工标注数据训练的结果，在剪辑准确率和句子 IoU 得分上超过 SOTA 方法分别达到 15.7 和 16.2 个百分点，证明了注释的有效性。同时，我们提供了六种具有不同多语言训练策略的基线方法，比较它们在单语内和跨语言设置中的性能，展示了多语言视频文本对齐的挑战。