We consider the problem of generating musical soundtracks in sync with
rhythmic visual cues. Most existing works rely on pre-defined music
representations, leading to the incompetence of generative flexibility and
complexity. Other methods directly generating video-conditioned waveforms
suffer from limited scenarios, short lengths, and unstable generation quality.
To this end, we present Long-Term Rhythmic Video Soundtracker (LORIS), a novel
framework to synthesize long-term conditional waveforms. Specifically, our
framework consists of a latent conditional diffusion probabilistic model to
perform waveform synthesis. Furthermore, a series of context-aware conditioning
encoders are proposed to take temporal information into consideration for a
long-term generation. Notably, we extend our model's applicability from dances
to multiple sports scenarios such as floor exercise and figure skating. To
perform comprehensive evaluations, we establish a benchmark for rhythmic video
soundtracks including the pre-processed dataset, improved evaluation metrics,
and robust generative baselines. Extensive experiments show that our model
generates long-term soundtracks with state-of-the-art musical quality and
rhythmic correspondence. Codes are available at
https://github.com/OpenGVLab/LORIS.

该论文介绍了一个称为 LORIS 的新框架，用于在长期条件下生成可靠的波形，以生成与节奏视觉提示同步的音乐配乐，并扩展其模型的适用性，从舞蹈到多种体育场景，如地面练习和花样滑冰。

长期节奏视频音轨器

Long-Term Rhythmic Video Soundtracker

We propose the multi-head convolutional neural network (MCNN) architecture
for waveform synthesis from spectrograms. Nonlinear interpolation in MCNN is
employed with transposed convolution layers in parallel heads. MCNN achieves
more than an order of magnitude higher compute intensity than commonly-used
iterative algorithms like Griffin-Lim, yielding efficient utilization for
modern multi-core processors, and very fast (more than 300x real-time) waveform
synthesis. For training of MCNN, we use a large-scale speech recognition
dataset and losses defined on waveforms that are related to perceptual audio
quality. We demonstrate that MCNN constitutes a very promising approach for
high-quality speech synthesis, without any iterative algorithms or
autoregression in computations.

本研究提出了一种使用多头卷积神经网络（MCNN）架构从频谱生成波形的方法，而不需要迭代算法，能显著提高计算效率。对 MCNN 进行了大规模数据集的训练，并且损失函数的定义关注于声学感知质量。证明了 MCNN 是一种非常有前途的高质量语音合成方法。

利用多头卷积神经网络实现快速谱图反演

Fast Spectrogram Inversion using Multi-head Convolutional Neural  Networks

We present Deep Voice 3, a fully-convolutional attention-based neural
text-to-speech (TTS) system. Deep Voice 3 matches state-of-the-art neural
speech synthesis systems in naturalness while training ten times faster. We
scale Deep Voice 3 to data set sizes unprecedented for TTS, training on more
than eight hundred hours of audio from over two thousand speakers. In addition,
we identify common error modes of attention-based speech synthesis networks,
demonstrate how to mitigate them, and compare several different waveform
synthesis methods. We also describe how to scale inference to ten million
queries per day on one single-GPU server.

Deep Voice 3 是一种全卷积自注意神经文本转语音系统，可以很好地匹配目前最先进的神经语音合成系统的自然度，同时训练速度快于十倍。通过在超过 2000 位演讲者的超过 800 小时的音频数据上进行训练，实现了 TTS 数据集规模的前所未有的扩展。同时，我们描述了如何缩放指向注意的语音合成网络的推断，以在单个 GPU 服务器上每天缩放到 1000 万个查询，并比较了几种不同的波形综合方法。