Benchmarking vision-based driving policies is challenging. On one hand,
open-loop evaluation with real data is easy, but these results do not reflect
closed-loop performance. On the other, closed-loop evaluation is possible in
simulation, but is hard to scale due to its significant computational demands.
Further, the simulators available today exhibit a large domain gap to real
data. This has resulted in an inability to draw clear conclusions from the
rapidly growing body of research on end-to-end autonomous driving. In this
paper, we present NAVSIM, a middle ground between these evaluation paradigms,
where we use large datasets in combination with a non-reactive simulator to
enable large-scale real-world benchmarking. Specifically, we gather
simulation-based metrics, such as progress and time to collision, by unrolling
bird's eye view abstractions of the test scenes for a short simulation horizon.
Our simulation is non-reactive, i.e., the evaluated policy and environment do
not influence each other. As we demonstrate empirically, this decoupling allows
open-loop metric computation while being better aligned with closed-loop
evaluations than traditional displacement errors. NAVSIM enabled a new
competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting
in several new insights. On a large set of challenging scenarios, we observe
that simple methods with moderate compute requirements such as TransFuser can
match recent large-scale end-to-end driving architectures such as UniAD. Our
modular framework can potentially be extended with new datasets, data curation
strategies, and metrics, and will be continually maintained to host future
challenges. Our code is available at
this https URL

在这篇论文中，我们介绍了一种介于开环评估和闭环评估之间的评估范式，即使用大型数据集与非反应式模拟器相结合，实现大规模真实世界基准测试。我们展示了该方法在衡量终到终自动驾驶时更好地与闭环评估对齐，并在 CVPR 2024 的新竞赛中取得了一些新的见解。