When editing a video, a piece of attractive background music is
indispensable. However, video background music generation tasks face several
challenges, for example, the lack of suitable training datasets, and the
difficulties in flexibly controlling the music generation process and
sequentially aligning the video and music. In this work, we first propose a
high-quality music-video dataset BGM909 with detailed annotation and shot
detection to provide multi-modal information about the video and music. We then
present evaluation metrics to assess music quality, including music diversity
and alignment between music and video with retrieval precision metrics.
Finally, we propose the Diff-BGM framework to automatically generate the
background music for a given video, which uses different signals to control
different aspects of the music during the generation process, i.e., uses
dynamic video features to control music rhythm and semantic features to control
the melody and atmosphere. We propose to align the video and music sequentially
by introducing a segment-aware cross-attention layer. Experiments verify the
effectiveness of our proposed method. The code and models are available at
this https URL

本文通过提出高质量的音乐视频数据集 BGM909 并进行详细注释和镜头检测，使用检索精确度度量标准来评估音乐质量，提出了 Diff-BGM 框架，以自动为给定的视频生成背景音乐，并引入片段感知交叉注意力层进行视频和音乐的顺序对齐，实验证明了我们提出方法的有效性。