Foley sound, audio content inserted synchronously with videos, plays a
critical role in the user experience of multimedia content. Recently, there has
been active research in Foley sound synthesis, leveraging the advancements in
deep generative models. However, such works mainly focus on replicating a
single sound class or a textual sound description, neglecting temporal
information, which is crucial in the practical applications of Foley sound. We
present T-Foley, a Temporal-event-guided waveform generation model for Foley
sound synthesis. T-Foley generates high-quality audio using two conditions: the
sound class and temporal event feature. For temporal conditioning, we devise a
temporal event feature and a novel conditioning technique named Block-FiLM.
T-Foley achieves superior performance in both objective and subjective
evaluation metrics and generates Foley sound well-synchronized with the
temporal events. Additionally, we showcase T-Foley's practical applications,
particularly in scenarios involving vocal mimicry for temporal event control.
We show the demo on our companion website.

T-Foley 是一种基于时间事件引导的波形生成模型，用于合成弗雷音效。该模型利用声音类别和时间事件特征生成高质量的音频，并且在客观和主观评估指标上显示出卓越的性能，能够与时间事件很好地同步生成 Foley 音效。

T-FOLEY：一种可控的波形领域扩散模型用于时间事件引导的福利音合成

T-FOLEY: A Controllable Waveform-Domain Diffusion Model for  Temporal-Event-Guided Foley Sound Synthesis

What does a typical visit to Paris look like? Do people first take photos of
the Louvre and then the Eiffel Tower? Can we visually model a temporal event
like "Paris Vacation" using current frameworks? In this paper, we explore how
we can automatically learn the temporal aspects, or storylines of visual
concepts from web data. Previous attempts focus on consecutive image-to-image
transitions and are unsuccessful at recovering the long-term underlying story.
Our novel Skipping Recurrent Neural Network (S-RNN) model does not attempt to
predict each and every data point in the sequence, like classic RNNs. Rather,
S-RNN uses a framework that skips through the images in the photo stream to
explore the space of all ordered subsets of the albums via an efficient
sampling procedure. This approach reduces the negative impact of strong
short-term correlations, and recovers the latent story more accurately. We show
how our learned storylines can be used to analyze, predict, and summarize photo
albums from Flickr. Our experimental results provide strong qualitative and
quantitative evidence that S-RNN is significantly better than other candidate
methods such as LSTMs on learning long-term correlations and recovering latent
storylines. Moreover, we show how storylines can help machines better
understand and summarize photo streams by inferring a brief personalized story
of each individual album.

本文介绍了 Skipping Recurrent Neural Network 模型，并应用于分析、预测和总结图像库。实验结果表明，与其他候选方法相比，S-RNN 在学习长期相关性和恢复潜在故事线方面表现显著优越，并展示了故事线如何帮助机器更好地理解和总结照片流