In recent years, video generation has become a prominent generative tool and
has drawn significant attention. However, there is little consideration in
audio-to-video generation, though audio contains unique qualities like temporal
semantics and magnitude. Hence, we propose The Power of Sound (TPoS) model to
incorporate audio input that includes both changeable temporal semantics and
magnitude. To generate video frames, TPoS utilizes a latent stable diffusion
model with textual semantic information, which is then guided by the sequential
audio embedding from our pretrained Audio Encoder. As a result, this method
produces audio reactive video contents. We demonstrate the effectiveness of
TPoS across various tasks and compare its results with current state-of-the-art
techniques in the field of audio-to-video generation. More examples are
available at this https URL

通过使用包含变化的时间语义和大小的音频输入的 The Power of Sound (TPoS) 模型结合文本语义信息和经过预训练的音频编码器的顺序音频嵌入，TPoS 方法能够生成对音频有反应的视频内容，并在各个任务中展示了其有效性，与当前音频到视频生成领域的最新技术进行了比较。

声音的力量：稳定扩散下的音频反应视频生成

The Power of Sound (TPoS): Audio Reactive Video Generation with Stable  Diffusion

Inspired by human neurological structures for action anticipation, we present
an action anticipation model that enables the prediction of plausible future
actions by forecasting both the visual and temporal future. In contrast to
current state-of-the-art methods which first learn a model to predict future
video features and then perform action anticipation using these features, the
proposed framework jointly learns to perform the two tasks, future visual and
temporal representation synthesis, and early action anticipation. The joint
learning framework ensures that the predicted future embeddings are informative
to the action anticipation task. Furthermore, through extensive experimental
evaluations we demonstrate the utility of using both visual and temporal
semantics of the scene, and illustrate how this representation synthesis could
be achieved through a recurrent Generative Adversarial Network (GAN) framework.
Our model outperforms the current state-of-the-art methods on multiple
datasets: UCF101, UCF101-24, UT-Interaction and TV Human Interaction.

本篇论文提出了一种行动预测模型，受人类神经结构的启发，在预测未来视觉和时间表现方面，能够预测合理的未来行动，并证明使用视觉和时间语义的场景表示结合递归生成对抗网络 (GAN) 框架可以实现未来表现的表示综合，该模型在多个数据集上优于当前最先进的方法。