The rise of multimodal large language models (MLLMs) has spurred interest in
language-based driving tasks. However, existing research typically focuses on
limited tasks and often omits key multi-view and temporal information which is
crucial for robust autonomous driving. To bridge these gaps, we introduce
NuInstruct, a novel dataset with 91K multi-view video-QA pairs across 17
subtasks, where each task demands holistic information (e.g., temporal,
multi-view, and spatial), significantly elevating the challenge level. To
obtain NuInstruct, we propose a novel SQL-based method to generate
instruction-response pairs automatically, which is inspired by the driving
logical progression of humans. We further present BEV-InMLLM, an end-to-end
method for efficiently deriving instruction-aware Bird's-Eye-View (BEV)
features, language-aligned for large language models. BEV-InMLLM integrates
multi-view, spatial awareness, and temporal semantics to enhance MLLMs'
capabilities on NuInstruct tasks. Moreover, our proposed BEV injection module
is a plug-and-play method for existing MLLMs. Our experiments on NuInstruct
demonstrate that BEV-InMLLM significantly outperforms existing MLLMs, e.g.
around 9% improvement on various tasks. We plan to release our NuInstruct for
future research development.

多模态大语言模型在自动驾驶任务中的应用，使用 NuInstruct 数据集和 BEV-InMLLM 方法进行多视角、时态相关信息的驾驶任务处理，相比现有的大语言模型，BEV-InMLLM 取得约 9% 的性能提升。