Objects' motions in nature are governed by complex interactions and their
properties. While some properties, such as shape and material, can be
identified via the object's visual appearances, others like mass and electric
charge are not directly visible. The compositionality between the visible and
hidden properties poses unique challenges for AI models to reason from the
physical world, whereas humans can effortlessly infer them with limited
observations. Existing studies on video reasoning mainly focus on visually
observable elements such as object appearance, movement, and contact
interaction. In this paper, we take an initial step to highlight the importance
of inferring the hidden physical properties not directly observable from visual
appearances, by introducing the Compositional Physical Reasoning (ComPhy)
dataset. For a given set of objects, ComPhy includes few videos of them moving
and interacting under different initial conditions. The model is evaluated
based on its capability to unravel the compositional hidden properties, such as
mass and charge, and use this knowledge to answer a set of questions posted on
one of the videos. Evaluation results of several state-of-the-art video
reasoning models on ComPhy show unsatisfactory performance as they fail to
capture these hidden properties. We further propose an oracle neural-symbolic
framework named Compositional Physics Learner (CPL), combining visual
perception, physical property learning, dynamic prediction, and symbolic
execution into a unified framework. CPL can effectively identify objects'
physical properties from their interactions and predict their dynamics to
answer questions.

本文介绍 ComPhy 数据集，该数据集含有一些移动和交互对象的视频，用于评估模型是否能够理解隐藏的物理性质，实验结果表明，一些当前先进的视频推理模型无法捕捉这些隐藏的属性。此外，本文进一步提出了一种名为 Compositional Physics Learner（CPL）的神经符号框架，该框架将视觉感知、物理特性学习、动态预测和符号执行结合成一个统一的框架，以识别对象的物理特性并预测它们的动态以回答问题。