We propose a hierarchical approach for making long-term predictions of future
frames. To avoid inherent compounding errors in recursive pixel-level
prediction, we propose to first estimate high-level structure in the input
frames, then predict how that structure evolves in the future, and finally by
observing a single frame from the past and the predicted high-level structure,
we construct the future frames without having to observe any of the pixel-level
predictions. Long-term video prediction is difficult to perform by recurrently
observing the predicted frames because the small errors in pixel space
exponentially amplify as predictions are made deeper into the future. Our
approach prevents pixel-level error propagation from happening by removing the
need to observe the predicted frames. Our model is built with a combination of
LSTM and analogy based encoder-decoder convolutional neural networks, which
independently predict the video structure and generate the future frames,
respectively. In experiments, our model is evaluated on the Human3.6M and Penn
Action datasets on the task of long-term pixel-level video prediction of humans
performing actions and demonstrate significantly better results than the
state-of-the-art.

本论文提出了一种层次化方法，以长期预测未来帧，通过预测输入帧中的高级结构，然后预测其未来的演变，并通过观察过去的单个帧和预测的高级结构来构建未来帧，提高了视频预测的效果，并在人类动作的长期像素级视频预测的任务上展现了较好的结果，超过了现有技术水平。