Rigorously testing autonomy systems is essential for making safe self-driving
vehicles (SDV) a reality. It requires one to generate safety critical scenarios
beyond what can be collected safely in the world, as many scenarios happen
rarely on public roads. To accurately evaluate performance, we need to test the
SDV on these scenarios in closed-loop, where the SDV and other actors interact
with each other at each timestep. Previously recorded driving logs provide a
rich resource to build these new scenarios from, but for closed loop
evaluation, we need to modify the sensor data based on the new scene
configuration and the SDV's decisions, as actors might be added or removed and
the trajectories of existing actors and the SDV will differ from the original
log. In this paper, we present UniSim, a neural sensor simulator that takes a
single recorded log captured by a sensor-equipped vehicle and converts it into
a realistic closed-loop multi-sensor simulation. UniSim builds neural feature
grids to reconstruct both the static background and dynamic actors in the
scene, and composites them together to simulate LiDAR and camera data at new
viewpoints, with actors added or removed and at new placements. To better
handle extrapolated views, we incorporate learnable priors for dynamic objects,
and leverage a convolutional network to complete unseen regions. Our
experiments show UniSim can simulate realistic sensor data with small domain
gap on downstream tasks. With UniSim, we demonstrate closed-loop evaluation of
an autonomy system on safety-critical scenarios as if it were in the real
world.

UniSim 是一个神经传感器模拟器，它可以将一个由传感器设备的车辆记录的日志转化为逼真的闭环多传感器模拟，用于安全关键场景的闭环评估。