With the emergence of Large Language Models (LLMs) and Vision Foundation Models (VFMs), multimodal AI systems benefiting from large models have the potential to equally perceive the real world, make decisions, and control tools as humans. In recent months, LLMs have shown widespread attention in autonomous driving and map systems. Despite its immense potential, there is still a lack of a comprehensive understanding of key challenges, opportunities, and future endeavors to apply in LLM driving systems. In this paper, we present a systematic investigation in this field. We first introduce the background of Multimodal Large Language Models (MLLMs), the multimodal models development using LLMs, and the history of autonomous driving. Then, we overview existing MLLM tools for driving, transportation, and map systems together with existing datasets and benchmarks. Moreover, we summarized the works in The 1st WACV Workshop on Large Language and Vision Models for Autonomous Driving (LLVM-AD), which is the first workshop of its kind regarding LLMs in autonomous driving. To further promote the development of this field, we also discuss several important problems regarding using MLLMs in autonomous driving systems that need to be solved by both academia and industry.

基于大型语言模型和视觉基础模型的多模态人工智能系统在实时感知、决策和工具控制方面与人类相似，并在自动驾驶和地图系统中引起广泛关注。本文对多模态大型语言模型驱动系统的关键挑战、机会和未来工作进行了系统研究，概述了多模态大型语言模型的背景、发展以及自动驾驶的历史，并总结了在自动驾驶中使用多模态大型语言模型的工具、数据集和基准，同时介绍了第一届大型语言和视觉模型自动驾驶研讨会的相关工作，提出了使用多模态大型语言模型在自动驾驶系统中需要解决的若干重要问题，并呼吁学术界和工业界共同促进该领域的发展。

自动驾驶的多模态大型语言模型调查