Accurately simulating real world object dynamics is essential for various
applications such as robotics, engineering, graphics, and design. To better
capture complex real dynamics such as contact and friction, learned simulators
based on graph networks have recently shown great promise. However, applying
these learned simulators to real scenes comes with two major challenges: first,
scaling learned simulators to handle the complexity of real world scenes which
can involve hundreds of objects each with complicated 3D shapes, and second,
handling inputs from perception rather than 3D state information. Here we
introduce a method which substantially reduces the memory required to run
graph-based learned simulators. Based on this memory-efficient simulation
model, we then present a perceptual interface in the form of editable NeRFs
which can convert real-world scenes into a structured representation that can
be processed by graph network simulator. We show that our method uses
substantially less memory than previous graph-based simulators while retaining
their accuracy, and that the simulators learned in synthetic environments can
be applied to real world scenes captured from multiple camera angles. This
paves the way for expanding the application of learned simulators to settings
where only perceptual information is available at inference time.

准确模拟真实世界物体动态对于机器人、工程、图形和设计等各种应用至关重要。我们提出了一种减少运行基于图网络的学习模拟器所需内存的方法。基于这种内存高效的模拟模型，我们提供了一种感知界面，以可编辑的 NeRFs 形式将真实世界场景转换为图网络模拟器可以处理的结构化表示。我们展示了我们的方法使用的内存远远少于以前基于图网络的模拟器同时保持其准确性，并且合成环境中学习的模拟器可以应用于从多个摄像机角度捕捉的真实世界场景，从而扩展了学习模拟器在仅在推断时有感知信息的设置中的应用。