Neural models are one of the most popular approaches for music generation,
yet there aren't standard large datasets tailored for learning music directly
from game data. To address this research gap, we introduce a novel dataset
named NES-VMDB, containing 98,940 gameplay videos from 389 NES games, each
paired with its original soundtrack in symbolic format (MIDI). NES-VMDB is
built upon the Nintendo Entertainment System Music Database (NES-MDB),
encompassing 5,278 music pieces from 397 NES games. Our approach involves
collecting long-play videos for 389 games of the original dataset, slicing them
into 15-second-long clips, and extracting the audio from each clip.
Subsequently, we apply an audio fingerprinting algorithm (similar to Shazam) to
automatically identify the corresponding piece in the NES-MDB dataset.
Additionally, we introduce a baseline method based on the Controllable Music
Transformer to generate NES music conditioned on gameplay clips. We evaluated
this approach with objective metrics, and the results showed that the
conditional CMT improves musical structural quality when compared to its
unconditional counterpart. Moreover, we used a neural classifier to predict the
game genre of the generated pieces. Results showed that the CMT generator can
learn correlations between gameplay videos and game genres, but further
research has to be conducted to achieve human-level performance.

通过引入一个名为 NES-VMDB 的新数据集，我们的研究填补了从游戏数据中直接学习音乐的标准大型数据集的研究空白，该数据集包含 389 个 NES 游戏的 98,940 个游戏视频，每个视频都与其原始的符号格式 (MIDI) 原声音轨配对。