Multimodal video-audio-text understanding and generation can benefit from
datasets that are narrow but rich. The narrowness allows bite-sized challenges
that the research community can make progress on. The richness ensures we are
making progress along the core challenges. To this end, we present a
large-scale video-audio-text dataset MUGEN, collected using the open-sourced
platform game CoinRun [11]. We made substantial modifications to make the game
richer by introducing audio and enabling new interactions. We trained RL agents
with different objectives to navigate the game and interact with 13 objects and
characters. This allows us to automatically extract a large collection of
diverse videos and associated audio. We sample 375K video clips (3.2s each) and
collect text descriptions from human annotators. Each video has additional
annotations that are extracted automatically from the game engine, such as
accurate semantic maps for each frame and templated textual descriptions.
Altogether, MUGEN can help progress research in many tasks in multimodal
understanding and generation. We benchmark representative approaches on tasks
involving video-audio-text retrieval and generation. Our dataset and code are
released at: this https URL.

本文介绍了通过引入音频和新的交互方式收集大量视频和相关音频的大型视频 - 音频 - 文本数据集 MUGEN，并展示了其在多模态理解和生成任务中的潜在应用。