Extracting and predicting object structure and dynamics from videos without
supervision is a major challenge in machine learning. To address this
challenge, we adopt a keypoint-based image representation and learn a
stochastic dynamics model of the keypoints. Future frames are reconstructed
from the keypoints and a reference frame. By modeling dynamics in the keypoint
coordinate space, we achieve stable learning and avoid compounding of errors in
pixel space. Our method improves upon unstructured representations both for
pixel-level video prediction and for downstream tasks requiring object-level
understanding of motion dynamics. We evaluate our model on diverse datasets: a
multi-agent sports dataset, the Human3.6M dataset, and datasets based on
continuous control tasks from the DeepMind Control Suite. The spatially
structured representation outperforms unstructured representations on a range
of motion-related tasks such as object tracking, action recognition and reward
prediction.

本研究针对无监督的视频对象结构和动态提取及预测的挑战，采用基于关键点的图像表示方法，并在关键点的动力学模型上进行学习，以实现像素空间中的稳定学习和错误复合的避免。该方法在像素级视频预测和需要物体级运动动态理解的下游任务方面均优于非结构化表示，并在多个数据集和任务中进行了评估和验证。