In cross-modal music processing, translation between visual, auditory, and
semantic content opens up new possibilities as well as challenges. The
construction of such a transformative scheme depends upon a benchmark corpus
with a comprehensive data infrastructure. In particular, the assembly of a
large-scale cross-modal dataset presents major challenges. In this paper, we
present the MOSA (Music mOtion with Semantic Annotation) dataset, which
contains high quality 3-D motion capture data, aligned audio recordings, and
note-by-note semantic annotations of pitch, beat, phrase, dynamic,
articulation, and harmony for 742 professional music performances by 23
professional musicians, comprising more than 30 hours and 570 K notes of data.
To our knowledge, this is the largest cross-modal music dataset with note-level
annotations to date. To demonstrate the usage of the MOSA dataset, we present
several innovative cross-modal music information retrieval (MIR) and musical
content generation tasks, including the detection of beats, downbeats, phrase,
and expressive contents from audio, video and motion data, and the generation
of musicians' body motion from given music audio. The dataset and codes are
available alongside this publication
(this https URL).

本文介绍了 MOSA（Music mOtion with Semantic Annotation）数据集，它包含由 23 位专业音乐家演奏的 742 个专业音乐表演的高质量三维运动捕捉数据，与音频录音及音高、节拍、乐句、动态、演奏和和谐等音符级的语义注释，是迄今为止最大的音乐领域的交互式数据集。数据集的用途在于交叉模态音乐信息检索（MIR）和音乐内容生成任务，包括从音频、视频和运动数据中检测节拍、下拍、乐句和表现内容，并根据给定的音乐音频生成音乐家的身体动作。数据集和代码可在本文所附网址中获取。