Realistic simulators are critical for training and verifying robotics
systems. While most of the contemporary simulators are hand-crafted, a
scaleable way to build simulators is to use machine learning to learn how the
environment behaves in response to an action, directly from data. In this work,
we aim to learn to simulate a dynamic environment directly in pixel-space, by
watching unannotated sequences of frames and their associated action pairs. We
introduce a novel high-quality neural simulator referred to as DriveGAN that
achieves controllability by disentangling different components without
supervision. In addition to steering controls, it also includes controls for
sampling features of a scene, such as the weather as well as the location of
non-player objects. Since DriveGAN is a fully differentiable simulator, it
further allows for re-simulation of a given video sequence, offering an agent
to drive through a recorded scene again, possibly taking different actions. We
train DriveGAN on multiple datasets, including 160 hours of real-world driving
data. We showcase that our approach greatly surpasses the performance of
previous data-driven simulators, and allows for new features not explored
before.

通过使用机器学习，该研究提出了一种能够通过观察图像数据和相关动作对来学习环境行为并进行建模的高质量神经模型，称为 DriveGAN，不需要监督信号训练控制模型，并在多个数据集上进行了训练和测试，表现优于之前的数据驱动模拟器，实现了对场景和非玩家对象的各个方面进行控制的新功能。