Current approaches in video forecasting attempt to generate videos directly in pixel space using Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs). However, since these approaches try to model all the structure and scene dynamics at once, in unconstrained settings they often generate uninterpretable results. Our insight is to model the forecasting problem at a higher level of abstraction. Specifically, we exploit human pose detectors as a free source of supervision and break the video forecasting problem into two discrete steps. First we explicitly model the high level structure of active objects in the scene---humans---and use a VAE to model the possible future movements of humans in the pose space. We then use the future poses generated as conditional information to a GAN to predict the future frames of the video in pixel space. By using the structured space of pose as an intermediate representation, we sidestep the problems that GANs have in generating video pixels directly. We show through quantitative and qualitative evaluation that our method outperforms state-of-the-art methods for video prediction.

该论文提出了一种基于高层抽象建模视频预测问题的方法，利用人体姿态探测器作为监督信号，将视频预测问题分解为两个离散步骤，明确建模场景中的高层次结构，使用变分自编码器(VAE)建模场景中活动对象的潜在未来运动，并将生成的未来姿势作为条件输入给生成对抗网络(GAN)预测未来视频帧，该方法通过呈现的定量和定性评估表明，其优于时下流行的视频预测方法。

姿态决定一切：通过生成姿态预测实现视频预测