A new trend in the computer vision community is to capture objects of
interest following flexible human command represented by a natural language
prompt. However, the progress of using language prompts in driving scenarios is
stuck in a bottleneck due to the scarcity of paired prompt-instance data. To
address this challenge, we propose the first object-centric language prompt set
for driving scenes within 3D, multi-view, and multi-frame space, named
NuPrompt. It expands Nuscenes dataset by constructing a total of 35,367
language descriptions, each referring to an average of 5.3 object tracks. Based
on the object-text pairs from the new benchmark, we formulate a new
prompt-based driving task, \ie, employing a language prompt to predict the
described object trajectory across views and frames. Furthermore, we provide a
simple end-to-end baseline model based on Transformer, named PromptTrack.
Experiments show that our PromptTrack achieves impressive performance on
NuPrompt. We hope this work can provide more new insights for the autonomous
driving community. Dataset and Code will be made public at
\href{https://github.com/wudongming97/Prompt4Driving}{this https URL}.

提出了第一个针对 3D、多视角和多帧空间中驾驶场景的以物体为中心的语言提示集 NuPrompt，并构建了包括 35,367 个语言描述的新的基准数据集。基于该数据集，提出了一种以语言提示为基础的驾驶任务，使用语言提示来预测描述物体在不同视角和帧中的轨迹，并提供了一个简单的基于 Transformer 的端到端基准模型 PromptTrack，实验结果表明 PromptTrack 在 NuPrompt 上取得了令人印象深刻的性能。希望这项工作能为自动驾驶社区提供更多的新见解。数据集和代码将在此 https URL 公开。