This paper presents GRASP, a novel benchmark to evaluate the language
grounding and physical understanding capabilities of video-based multimodal
large language models (LLMs). This evaluation is accomplished via a two-tier
approach leveraging Unity simulations. The initial level tests for language
grounding by assessing a model's ability to relate simple textual descriptions
with visual information. The second level evaluates the model's understanding
of 'Intuitive Physics' principles, such as object permanence and continuity. In
addition to releasing the benchmark, we use it to evaluate several
state-of-the-art multimodal LLMs. Our evaluation reveals significant
shortcomings in current models' language grounding and intuitive physics. These
identified limitations underline the importance of benchmarks like GRASP to
monitor the progress of future models in developing these competencies.

该研究提出 GRASP，这是一个用于评估基于视频多模式大型语言模型（LLMs）的语言理解和物理理解能力的新的基准。通过利用 Unity 模拟技术，这一评估通过两级方法完成。初始级别测试语言理解能力，通过评估模型将简单文本描述与视觉信息联系起来的能力。第二级别评估模型对直观物理原理（如物体的永恒性和连续性）的理解能力。除了发布基准系统，研究还使用它来评估几种最新的多模式 LLMs。我们的评估揭示了当前模型在语言理解和直观物理方面的重大缺陷。这些发现的限制凸显了像 GRASP 这样的基准对于监测未来模型在这些能力上的进展的重要性。

GRASP: 用于评估多模态语言模型中语言引用与情境物理理解的新型基准测试

GRASP: A novel benchmark for evaluating language GRounding And Situated  Physics understanding in multimodal language models

As children grow older, they develop an intuitive understanding of the
physical processes around them. Their physical understanding develops in
stages, moving along developmental trajectories which have been mapped out
extensively in previous empirical research. Here, we investigate how the
learning trajectories of deep generative neural networks compare to children's
developmental trajectories using physical understanding as a testbed. We
outline an approach that allows us to examine two distinct hypotheses of human
development - stochastic optimization and complexity increase. We find that
while our models are able to accurately predict a number of physical processes,
their learning trajectories under both hypotheses do not follow the
developmental trajectories of children.

根据物理理解为测试基准，通过深度生成神经网络的学习轨迹与儿童的发展轨迹进行比较，研究发现模型在两种假设下的学习轨迹均不符合儿童的发展轨迹。

生成神经网络中的物理知识获取

The Acquisition of Physical Knowledge in Generative Neural Networks

As children grow older, they develop an intuitive understanding of the
physical processes around them. They move along developmental trajectories,
which have been mapped out extensively in previous empirical research. We
investigate how children's developmental trajectories compare to the learning
trajectories of artificial systems. Specifically, we examine the idea that
cognitive development results from some form of stochastic optimization
procedure. For this purpose, we train a modern generative neural network model
using stochastic gradient descent. We then use methods from the developmental
psychology literature to probe the physical understanding of this model at
different degrees of optimization. We find that the model's learning trajectory
captures the developmental trajectories of children, thereby providing support
to the idea of development as stochastic optimization.

研究发现，神经网络的学习轨迹可以捕捉儿童认知发展的轨迹，从而支持将认知发展解释为某种随机优化过程的观点。

随机梯度下降揭示儿童学习物理的过程

Stochastic Gradient Descent Captures How Children Learn About Physics

In this paper, we study the challenging problem of predicting the dynamics of
objects in static images. Given a query object in an image, our goal is to
provide a physical understanding of the object in terms of the forces acting
upon it and its long term motion as response to those forces. Direct and
explicit estimation of the forces and the motion of objects from a single image
is extremely challenging. We define intermediate physical abstractions called
Newtonian scenarios and introduce Newtonian Neural Network ($N^3$) that learns
to map a single image to a state in a Newtonian scenario. Our experimental
evaluations show that our method can reliably predict dynamics of a query
object from a single image. In addition, our approach can provide physical
reasoning that supports the predicted dynamics in terms of velocity and force
vectors. To spur research in this direction we compiled Visual Newtonian
Dynamics (VIND) dataset that includes 6806 videos aligned with Newtonian
scenarios represented using game engines, and 4516 still images with their
ground truth dynamics.

该研究介绍了一种名为 Newtonian Neural Network 的中间物理抽象概念，可以从静态图像中学习物体的动态运动以及感受上的力。通过新构建的 Visual Newtonian Dynamics 数据集测试，显示该方法能够可靠地从单个图像中预测查询物体的动态，并支持以速度和力向量的方式进行预测的物理推理。