We consider the problem of generating musical soundtracks in sync with
rhythmic visual cues. Most existing works rely on pre-defined music
representations, leading to the incompetence of generative flexibility and
complexity. Other methods directly generating video-conditioned waveforms
suffer from limited scenarios, short lengths, and unstable generation quality.
To this end, we present Long-Term Rhythmic Video Soundtracker (LORIS), a novel
framework to synthesize long-term conditional waveforms. Specifically, our
framework consists of a latent conditional diffusion probabilistic model to
perform waveform synthesis. Furthermore, a series of context-aware conditioning
encoders are proposed to take temporal information into consideration for a
long-term generation. Notably, we extend our model's applicability from dances
to multiple sports scenarios such as floor exercise and figure skating. To
perform comprehensive evaluations, we establish a benchmark for rhythmic video
soundtracks including the pre-processed dataset, improved evaluation metrics,
and robust generative baselines. Extensive experiments show that our model
generates long-term soundtracks with state-of-the-art musical quality and
rhythmic correspondence. Codes are available at
https://github.com/OpenGVLab/LORIS.

该论文介绍了一个称为 LORIS 的新框架，用于在长期条件下生成可靠的波形，以生成与节奏视觉提示同步的音乐配乐，并扩展其模型的适用性，从舞蹈到多种体育场景，如地面练习和花样滑冰。