Understanding causes and effects in mechanical systems is an essential component of reasoning in the physical world. This work poses a new problem of counterfactual learning of object mechanics from visual input. We develop the COPHY benchmark to assess the capacity of the state-of-the-art models for causal physical reasoning in a synthetic 3D environment and propose a model for learning the physical dynamics in a counterfactual setting. Having observed a mechanical experiment that involves, for example, a falling tower of blocks, a set of bouncing balls or colliding objects, we learn to predict how its outcome is affected by an arbitrary intervention on its initial conditions, such as displacing one of the objects in the scene. The alternative future is predicted given the altered past and a latent representation of the confounders learned by the model in an end-to-end fashion with no supervision. We compare against feedforward video prediction baselines and show how observing alternative experiences allows the network to capture latent physical properties of the environment, which results in significantly more accurate predictions at the level of super human performance.

研究提出了一个新的问题，即如何从视觉输入中进行物体力学的反事实学习，并提出了一种用于在反事实情境下学习物理动态的模型。通过学习对初始条件进行干预时对其结果的影响，和对潜在因素的学习，该网络可以捕捉到物理环境的潜在特性，从而具有超越人类表现的预测能力。

CoPhy: 计算物理动力学的反事实学习