Summarization of multimedia data becomes increasingly significant as it is
the basis for many real-world applications, such as question answering, Web
search, and so forth. Most existing multi-modal summarization works however
have used visual complementary features extracted from images rather than
videos, thereby losing abundant information. Hence, we propose a novel
multi-modal summarization task to summarize from a document and its associated
video. In this work, we also build a baseline general model with effective
strategies, i.e., bi-hop attention and improved late fusion mechanisms to
bridge the gap between different modalities, and a bi-stream summarization
strategy to employ text and video summarization simultaneously. Comprehensive
experiments show that the proposed model is beneficial for multi-modal
summarization and superior to existing methods. Moreover, we collect a novel
dataset and it provides a new resource for future study that results from
documents and videos.

本文提出了一种基于文档和相关视频的新型多模态摘要任务，并构建了一个基于 bi-hop attention 和改进的 late fusion 机制的双流摘要模型，旨在同时处理文本和视频摘要。实验结果表明，该模型有利于多模态摘要且优于现有方法，同时构建了一个新的文档和视频数据集作为未来研究的资源。