Despite significant recent progress in the field of autonomous driving,
modern methods still struggle and can incur serious accidents when encountering
long-tail unforeseen events and challenging urban scenarios. On the one hand,
large language models (LLM) have shown impressive reasoning capabilities that
approach "Artificial General Intelligence". On the other hand, previous
autonomous driving methods tend to rely on limited-format inputs (e.g. sensor
data and navigation waypoints), restricting the vehicle's ability to understand
language information and interact with humans. To this end, this paper
introduces LMDrive, a novel language-guided, end-to-end, closed-loop autonomous
driving framework. LMDrive uniquely processes and integrates multi-modal sensor
data with natural language instructions, enabling interaction with humans and
navigation software in realistic instructional settings. To facilitate further
research in language-based closed-loop autonomous driving, we also publicly
release the corresponding dataset which includes approximately 64K
instruction-following data clips, and the LangAuto benchmark that tests the
system's ability to handle complex instructions and challenging driving
scenarios. Extensive closed-loop experiments are conducted to demonstrate
LMDrive's effectiveness. To the best of our knowledge, we're the very first
work to leverage LLMs for closed-loop end-to-end autonomous driving. Codes can
be found at this https URL

本文介绍了 LMDrive，一种语言引导的、端到端的、闭环自动驾驶框架，它独特地处理和整合多模态传感器数据和自然语言指令，实现了与人类和导航软件在真实教学环境中的交互。为了促进基于语言的闭环自动驾驶的进一步研究，我们还公开发布了包含约 64K 个指令跟踪数据片段的数据集，以及测试系统处理复杂指令和具有挑战性驾驶场景的 LangAuto 基准。进行了广泛的闭环实验以证明 LMDrive 的有效性。据我们所知，我们是首个利用 LGM 进行闭环端到端自动驾驶的工作。相关代码可在此网址找到：https://