We provide a sober look at the application of Multimodal Large Language
Models (MLLMs) within the domain of autonomous driving and challenge/verify
some common assumptions, focusing on their ability to reason and interpret
dynamic driving scenarios through sequences of images/frames in a closed-loop
control environment. Despite the significant advancements in MLLMs like GPT-4V,
their performance in complex, dynamic driving environments remains largely
untested and presents a wide area of exploration. We conduct a comprehensive
experimental study to evaluate the capability of various MLLMs as world models
for driving from the perspective of a fixed in-car camera. Our findings reveal
that, while these models proficiently interpret individual images, they
struggle significantly with synthesizing coherent narratives or logical
sequences across frames depicting dynamic behavior. The experiments demonstrate
considerable inaccuracies in predicting (i) basic vehicle dynamics
(forward/backward, acceleration/deceleration, turning right or left), (ii)
interactions with other road actors (e.g., identifying speeding cars or heavy
traffic), (iii) trajectory planning, and (iv) open-set dynamic scene reasoning,
suggesting biases in the models' training data. To enable this experimental
study we introduce a specialized simulator, DriveSim, designed to generate
diverse driving scenarios, providing a platform for evaluating MLLMs in the
realms of driving. Additionally, we contribute the full open-source code and a
new dataset, "Eval-LLM-Drive", for evaluating MLLMs in driving. Our results
highlight a critical gap in the current capabilities of state-of-the-art MLLMs,
underscoring the need for enhanced foundation models to improve their
applicability in real-world dynamic environments.

我们评估了多模态大型语言模型（MLLMs）在自动驾驶领域的应用，并挑战和验证了一些常见假设，重点关注它们在封闭环控制环境下通过图像 / 帧序列来推理和解释动态行驶情景的能力。我们的研究揭示了这些模型在预测复杂、动态的驾驶环境中存在的不足，对于描绘动态行为的帧之间能否综合连贯的叙述或逻辑序列存在很大困难。我们通过使用专门设计的模拟器 DriveSim 生成多样化的驾驶情景，进行了一项全面的实验研究来评估各种 MLLMs 作为驾驶世界模型的能力，并为评估驾驶中的 MLLMs 贡献了全面的开源代码和一个新数据集 “Eval-LLM-Drive”。我们的研究结果突出了当前领先的 MLLMs 在真实动态环境中应用能力上的重要差距，强调了需要改进基础模型以提高其在真实世界动态环境中的适用性。

探究多模态 LLMs 作为驾驶世界模型

Probing Multimodal LLMs as World Models for Driving

A longstanding challenge for self-driving development is simulating dynamic
driving scenarios seeded from recorded driving logs. In pursuit of this
functionality, we apply tools from discrete sequence modeling to model how
vehicles, pedestrians and cyclists interact in driving scenarios. Using a
simple data-driven tokenization scheme, we discretize trajectories to
centimeter-level resolution using a small vocabulary. We then model the
multi-agent sequence of motion tokens with a GPT-like encoder-decoder that is
autoregressive in time and takes into account intra-timestep interaction
between agents. Scenarios sampled from our model exhibit state-of-the-art
realism; our model tops the Waymo Sim Agents Benchmark, surpassing prior work
along the realism meta metric by 3.3% and along the interaction metric by 9.9%.
We ablate our modeling choices in full autonomy and partial autonomy settings,
and show that the representations learned by our model can quickly be adapted
to improve performance on nuScenes. We additionally evaluate the scalability of
our model with respect to parameter count and dataset size, and use density
estimates from our model to quantify the saliency of context length and
intra-timestep interaction for the traffic modeling task.

通过离散序列建模方法，我们研究了自动驾驶开发中模拟基于记录驾驶日志的动态驾驶场景的挑战。使用简单的数据驱动分词方案，将轨迹离散化到厘米级分辨率，并采用类似 GPT 模型的编码器 - 解码器来建模车辆、行人和骑车人在驾驶场景中的互动。通过在模型中抽样场景，我们展示了该模型具有最先进的真实性，且在 Waymo Sim Agents Benchmark 上超越先前的工作，其中真实性值成功提高了 3.3%，与其他模型相比，互动度提高了 9.9%。我们还在完全自动驾驶和部分自动驾驶环境中分析了我们的建模选择，并展示了我们的模型学到的表示可以快速适用于改进 nuScenes 的性能。此外，我们还对模型的参数数量和数据集规模进行了可扩展性评估，并利用模型得出的密度估计量化了上下文长度和时间间隔内交互对于交通建模任务的重要性。

轨迹英语：学习驾驶场景的语言

Trajeglish: Learning the Language of Driving Scenarios

Realistic and diverse traffic scenarios in large quantities are crucial for
the development and validation of autonomous driving systems. However, owing to
numerous difficulties in the data collection process and the reliance on
intensive annotations, real-world datasets lack sufficient quantity and
diversity to support the increasing demand for data. This work introduces
DriveSceneGen, a data-driven driving scenario generation method that learns
from the real-world driving dataset and generates entire dynamic driving
scenarios from scratch. DriveSceneGen is able to generate novel driving
scenarios that align with real-world data distributions with high fidelity and
diversity. Experimental results on 5k generated scenarios highlight the
generation quality, diversity, and scalability compared to real-world datasets.
To the best of our knowledge, DriveSceneGen is the first method that generates
novel driving scenarios involving both static map elements and dynamic traffic
participants from scratch.

大规模多样化的真实交通场景对于自动驾驶系统的开发和验证至关重要。本研究引入了 DriveSceneGen，一种数据驱动的驾驶场景生成方法，从真实世界的驾驶数据集中学习并生成完整的动态驾驶场景。与真实世界的数据集相比，DriveSceneGen 能够以高保真度和多样性生成与实际数据分布相吻合的新型驾驶场景。与真实世界数据集相比，5,000 个生成场景的实验结果突出了生成质量、多样性和可扩展性。据我们所知，DriveSceneGen 是首个能够从头开始生成涉及静态地图元素和动态交通参与者的新型驾驶场景的技术。