Generative models trained on internet data have revolutionized how text,
image, and video content can be created. Perhaps the next milestone for
generative models is to simulate realistic experience in response to actions
taken by humans, robots, and other interactive agents. Applications of a
real-world simulator range from controllable content creation in games and
movies, to training embodied agents purely in simulation that can be directly
deployed in the real world. We explore the possibility of learning a universal
simulator (UniSim) of real-world interaction through generative modeling. We
first make the important observation that natural datasets available for
learning a real-world simulator are often rich along different axes (e.g.,
abundant objects in image data, densely sampled actions in robotics data, and
diverse movements in navigation data). With careful orchestration of diverse
datasets, each providing a different aspect of the overall experience, UniSim
can emulate how humans and agents interact with the world by simulating the
visual outcome of both high-level instructions such as "open the drawer" and
low-level controls such as "move by x, y" from otherwise static scenes and
objects. There are numerous use cases for such a real-world simulator. As an
example, we use UniSim to train both high-level vision-language planners and
low-level reinforcement learning policies, each of which exhibit zero-shot
real-world transfer after training purely in a learned real-world simulator. We
also show that other types of intelligence such as video captioning models can
benefit from training with simulated experience in UniSim, opening up even
wider applications. Video demos can be found at
this https URL

通过生成模型学习一个真实世界互动的通用模拟器 (UniSim)，以模拟高级指令和低级控制的可视结果，并用于训练高级视觉 - 语言规划器和低级增强学习策略，实现纯粹从学习的真实世界模拟器中的零样本现实世界转移。