Lecture slide presentations, a sequence of pages that contain text and
figures accompanied by speech, are constructed and presented carefully in order
to optimally transfer knowledge to students. Previous studies in multimedia and
psychology attribute the effectiveness of lecture presentations to their
multimodal nature. As a step toward developing AI to aid in student learning as
intelligent teacher assistants, we introduce the Multimodal Lecture
Presentations dataset as a large-scale benchmark testing the capabilities of
machine learning models in multimodal understanding of educational content. Our
dataset contains aligned slides and spoken language, for 180+ hours of video
and 9000+ slides, with 10 lecturers from various subjects (e.g., computer
science, dentistry, biology). We introduce two research tasks which are
designed as stepping stones towards AI agents that can explain (automatically
captioning a lecture presentation) and illustrate (synthesizing visual figures
to accompany spoken explanations) educational content. We provide manual
annotations to help implement these two research tasks and evaluate
state-of-the-art models on them. Comparing baselines and human student
performances, we find that current models struggle in (1) weak crossmodal
alignment between slides and spoken text, (2) learning novel visual mediums,
(3) technical language, and (4) long-range sequences. Towards addressing this
issue, we also introduce PolyViLT, a multimodal transformer trained with a
multi-instance learning loss that is more effective than current approaches. We
conclude by shedding light on the challenges and opportunities in multimodal
understanding of educational presentations.

介绍了一个为了测试机器学习模型在多模态教育内容理解方面表现的新的数据集 ——Multimodal Lecture Presentations 数据集，并且提出了两个任务 —— 解释和说明教育内容，其中前者实现语音识别，后者实现视觉内容合成。介绍了一个名为 PolyViLT 的多模态变形器，该模型采用多个实例学习方法比现有方法更有效，同时指出了在多模态教育表现方面的挑战和机遇。