Publishing open-source academic video recordings is an emergent and prevalent
approach to sharing knowledge online. Such videos carry rich multimodal
information including speech, the facial and body movements of the speakers, as
well as the texts and pictures in the slides and possibly even the papers.
Although multiple academic video datasets have been constructed and released,
few of them support both multimodal content recognition and understanding
tasks, which is partially due to the lack of high-quality human annotations. In
this paper, we propose a novel multimodal, multigenre, and multipurpose
audio-visual academic lecture dataset (M$^3$AV), which has almost 367 hours of
videos from five sources covering computer science, mathematics, and medical
and biology topics. With high-quality human annotations of the spoken and
written words, in particular high-valued name entities, the dataset can be used
for multiple audio-visual recognition and understanding tasks. Evaluations
performed on contextual speech recognition, speech synthesis, and slide and
script generation tasks demonstrate that the diversity of M$^3$AV makes it a
challenging dataset.

通过提供高质量的人工注释，本文提出了一个全新的多模态、多类型和多用途的音频 - 视频学术讲座数据集，旨在支持多种音视频识别和理解任务，从而展示了 M^3AV 数据集的多样性和挑战性。