Multimodal summarization with multimodal output (MSMO) has emerged as a
promising research direction. Nonetheless, numerous limitations exist within
existing public MSMO datasets, including insufficient upkeep, data
inaccessibility, limited size, and the absence of proper categorization, which
pose significant challenges to effective research. To address these challenges
and provide a comprehensive dataset for this new direction, we have
meticulously curated the MultiSum dataset. Our new dataset features (1)
Human-validated summaries for both video and textual content, providing
superior human instruction and labels for multimodal learning. (2)
Comprehensively and meticulously arranged categorization, spanning 17 principal
categories and 170 subcategories to encapsulate a diverse array of real-world
scenarios. (3) Benchmark tests performed on the proposed dataset to assess
varied tasks and methods, including video temporal segmentation, video
summarization, text summarization, and multimodal summarization. To champion
accessibility and collaboration, we release the MultiSum dataset and the data
collection tool as fully open-source resources, fostering transparency and
accelerating future developments. Our project website can be found at
this https URL

通过创建包含视频和文本内容的人工验证摘要、17 个主分类和 170 个子分类的全面的数据集 MultiSum dataset，本文针对现有公共 MSMO 数据集的限制进行研究，进行了基准测试，并推出了数据收集工具和开放源代码资源以促进透明度和加速未来发展。