We present a VAE architecture for encoding and generating high dimensional sequential data, such as video or audio. Our deep generative model learns a latent representation of the data which is split into a static and dynamic part, allowing us to approximately disentangle latent time-dependent features (dynamics) from features which are preserved over time (content). This architecture gives us partial control over generating content and dynamics by conditioning on either one of these sets of features. In our experiments on artificially generated cartoon video clips and voice recordings, we show that we can convert the content of a given sequence into another one by such content swapping. For audio, this allows us to convert a male speaker into a female speaker and vice versa, while for video we can separately manipulate shapes and dynamics. Furthermore, we give empirical evidence for the hypothesis that stochastic RNNs as latent state models are more efficient at compressing and generating long sequences than deterministic ones, which may be relevant for applications in video compression.

这篇论文介绍了一种针对高维时序数据（如视频或音频）的 VAE 模型架构，学习了数据的潜在表示，可以近似地分解潜在的时间相关特征（动态）和随时间保持不变的特征（内容），从而进行内容和动态的部分控制，并给出了在人工生成的卡通视频片段和声音记录上的实验证据，进一步论证了随机 RNN 模型相对于确定性 RNN 对长序列的压缩与生成的效率更高的假设。

离散序列自编码器