Autonomous driving systems are increasingly popular in today's technological
landscape, where vehicles with partial automation have already been widely
available on the market, and the full automation era with ``driverless''
capabilities is near the horizon. However, accurately understanding humans'
commands, particularly for autonomous vehicles that have only passengers
instead of drivers, and achieving a high level of personalization remain
challenging tasks in the development of autonomous driving systems. In this
paper, we introduce a Large Language Model (LLM)-based framework Talk-to-Drive
(Talk2Drive) to process verbal commands from humans and make autonomous driving
decisions with contextual information, satisfying their personalized
preferences for safety, efficiency, and comfort. First, a speech recognition
module is developed for Talk2Drive to interpret verbal inputs from humans to
textual instructions, which are then sent to LLMs for reasoning. Then,
appropriate commands for the Electrical Control Unit (ECU) are generated,
achieving a 100\% success rate in executing codes. Real-world experiments show
that our framework can substantially reduce the takeover rate for a diverse
range of drivers by up to 90.1\%. To the best of our knowledge, Talk2Drive
marks the first instance of employing an LLM-based system in a real-world
autonomous driving environment.

利用大型语言模型（LLM）的 Talk-to-Drive 框架，通过处理人类的口头指令并结合上下文信息做出自主驾驶决策，实现个性化的安全、高效和舒适，成功率达到 100% 的执行命令，并在现实世界的实验中将不同驾驶员的接管率大幅降低至高达 90.1%，是在真实自主驾驶环境中应用 LLM 的首次实例。