We propose a model that is able to perform unsupervised physical parameter
estimation of systems from video, where the differential equations governing
the scene dynamics are known, but labeled states or objects are not available.
Existing physical scene understanding methods require either object state
supervision, or do not integrate with differentiable physics to learn
interpretable system parameters and states. We address this problem through a
physics-as-inverse-graphics approach that brings together
vision-as-inverse-graphics and differentiable physics engines, enabling objects
and explicit state and velocity representations to be discovered. This
framework allows us to perform long term extrapolative video prediction, as
well as vision-based model-predictive control. Our approach significantly
outperforms related unsupervised methods in long-term future frame prediction
of systems with interacting objects (such as ball-spring or 3-body
gravitational systems), due to its ability to build dynamics into the model as
an inductive bias. We further show the value of this tight vision-physics
integration by demonstrating data-efficient learning of vision-actuated
model-based control for a pendulum system. We also show that the controller's
interpretability provides unique capabilities in goal-driven control and
physical reasoning for zero-data adaptation.

本文提出了一种基于物理 - 逆向图像学的方法，可以从视频中无监督地估算系统的物理参数，实现长期视频预测和基于视觉的模型预测控制，并在交互物体系统的长期未来帧预测中显著优于相关无监督方法，还实现了控制器的可解释性提供了独特的能力在目标驱动控制和物理推理方面的零数据适应。