Due to the impressive capabilities of multimodal large language models
(MLLMs), recent works have focused on employing MLLM-based agents for
autonomous driving in large-scale and dynamic environments. However, prevalent
approaches often directly translate high-level instructions into low-level
vehicle control signals, which deviates from the inherent language generation
paradigm of MLLMs and fails to fully harness their emergent powers. As a
result, the generalizability of these methods is highly restricted by
autonomous driving datasets used during fine-tuning. To tackle this challenge,
we propose to connect high-level instructions and low-level control signals
with mid-level language-driven commands, which are more fine-grained than
high-level instructions but more universal and explainable than control
signals, and thus can effectively bridge the gap in between. We implement this
idea through a hierarchical multi-agent driving system named AD-H, including a
MLLM planner for high-level reasoning and a lightweight controller for
low-level execution. The hierarchical design liberates the MLLM from low-level
control signal decoding and therefore fully releases their emergent capability
in high-level perception, reasoning, and planning. We build a new dataset with
action hierarchy annotations. Comprehensive closed-loop evaluations demonstrate
several key advantages of our proposed AD-H system. First, AD-H can notably
outperform state-of-the-art methods in achieving exceptional driving
performance, even exhibiting self-correction capabilities during vehicle
operation, a scenario not encountered in the training dataset. Second, AD-H
demonstrates superior generalization under long-horizon instructions and novel
environmental conditions, significantly surpassing current state-of-the-art
methods. We will make our data and code publicly accessible at
this https URL

通过使用多模态大型语言模型（MLLMs）实现自动驾驶，连接高级指令和低级控制信号，形成多层次驱动系统 AD-H 提供了优秀的驾驶性能和普适性，充分发挥了 MLLM 在感知、推理和规划方面的潜力。