We describe a first step towards learning general-purpose visual
representations of physical scenes using only image prediction as a training
criterion. To do so, we first define "physical scene" and show that, even
though different agents may maintain different representations of the same
scene, the underlying physical scene that can be inferred is unique. Then, we
show that NeRFs cannot represent the physical scene, as they lack extrapolation
mechanisms. Those, however, could be provided by Diffusion Models, at least in
theory. To test this hypothesis empirically, NeRFs can be combined with
Diffusion Models, a process we refer to as NeRF Diffusion, used as unsupervised
representations of the physical scene. Our analysis is limited to visual data,
without external grounding mechanisms that can be provided by independent
sensory modalities.

通过图像预测作为训练标准，我们探索了学习物理场景通用视觉表示的第一步。我们定义了 “物理场景”，发现即使不同的代理可能维护同一场景的不同表示，可以推断出的基本物理场景是唯一的。同时，我们显示了 NeRF 不能代表物理场景，因为它们缺乏外推机制。然而，这种机制在理论上可以由扩散模型提供。为了通过实验证明这个假设，我们可以将 NeRF 和 Diffusion Models 结合使用，这个过程被我们称为 NeRF Diffusion，用于表示物理场景的无监督。我们的分析仅限于视觉数据，没有额外的基于独立感觉模态的基础机制提供。

构建物理场景的视觉基础模型

Towards Visual Foundational Models of Physical Scenes

Understanding and interacting with everyday physical scenes requires rich
knowledge about the structure of the world, represented either implicitly in a
value or policy function, or explicitly in a transition model. Here we
introduce a new class of learnable models--based on graph networks--which
implement an inductive bias for object- and relation-centric representations of
complex, dynamical systems. Our results show that as a forward model, our
approach supports accurate predictions from real and simulated data, and
surprisingly strong and efficient generalization, across eight distinct
physical systems which we varied parametrically and structurally. We also found
that our inference model can perform system identification. Our models are also
differentiable, and support online planning via gradient-based trajectory
optimization, as well as offline policy optimization. Our framework offers new
opportunities for harnessing and exploiting rich knowledge about the world, and
takes a key step toward building machines with more human-like representations
of the world.

介绍了一种基于图网络的学习模型，实现了对象和关系为中心的表示的归纳偏差，从而能够更准确地预测复杂动态系统的行为，并支持在线与离线规划。