Learning to predict the long-term future of video frames is notoriously
challenging due to inherent ambiguities in the distant future and dramatic
amplifications of prediction error through time. Despite the recent advances in
the literature, existing approaches are limited to moderately short-term
prediction (less than a few seconds), while extrapolating it to a longer future
quickly leads to destruction in structure and content. In this work, we revisit
hierarchical models in video prediction. Our method predicts future frames by
first estimating a sequence of semantic structures and subsequently translating
the structures to pixels by video-to-video translation. Despite the simplicity,
we show that modeling structures and their dynamics in the discrete semantic
structure space with a stochastic recurrent estimator leads to surprisingly
successful long-term prediction. We evaluate our method on three challenging
datasets involving car driving and human dancing, and demonstrate that it can
generate complicated scene structures and motions over a very long time horizon
(i.e., thousands frames), setting a new standard of video prediction with
orders of magnitude longer prediction time than existing approaches. Full
videos and codes are available at this https URL

本文重新审视了视频预测中的分层模型，通过先估计语义结构序列，再通过视频到视频的转换将结构转化为像素，我们展示了用随机循环估算器模拟离散语义结构空间中的结构及其动态来进行成功的长期预测，通过在汽车驾驶和人类舞蹈等三个数据集上的评估，证明了我们的方法能够在非常长的时间范围内生成复杂的场景结构和运动，并且取得了比现有方法长几数量级的预测时间，代码和完整视频可在此 https URL 获得。