Large Language Models (LLMs) have shown promise in the autonomous driving
sector, particularly in generalization and interpretability. We introduce a
unique object-level multimodal LLM architecture that merges vectorized numeric
modalities with a pre-trained LLM to improve context understanding in driving
situations. We also present a new dataset of 160k QA pairs derived from 10k
driving scenarios, paired with high quality control commands collected with RL
agent and question answer pairs generated by teacher LLM (GPT-3.5). A distinct
pretraining strategy is devised to align numeric vector modalities with static
LLM representations using vector captioning language data. We also introduce an
evaluation metric for Driving QA and demonstrate our LLM-driver's proficiency
in interpreting driving scenarios, answering questions, and decision-making.
Our findings highlight the potential of LLM-based driving action generation in
comparison to traditional behavioral cloning. We make our benchmark, datasets,
and model available for further exploration.

基于大型语言模型的自动驾驶行动生成方法在泛化性能和可解释性方面显示出潜力，提出了一种独特的对象级多模态语言模型体系结构，将向量化数值模态与预训练的语言模型相结合，以提高驾驶情境的上下文理解能力，通过新的数据集和评估度量，展示了该方法在解释驾驶情境、回答问题和决策等方面的能力。