Recent advancements in foundation models (FMs) have unlocked new prospects in
autonomous driving, yet the experimental settings of these studies are
preliminary, over-simplified, and fail to capture the complexity of real-world
driving scenarios in human environments. It remains under-explored whether FM
agents can handle long-horizon navigation tasks with free-from dialogue and
deal with unexpected situations caused by environmental dynamics or task
changes. To explore the capabilities and boundaries of FMs faced with the
challenges above, we introduce DriVLMe, a video-language-model-based agent to
facilitate natural and effective communication between humans and autonomous
vehicles that perceive the environment and navigate. We develop DriVLMe from
both embodied experiences in a simulated environment and social experiences
from real human dialogue. While DriVLMe demonstrates competitive performance in
both open-loop benchmarks and closed-loop human studies, we reveal several
limitations and challenges, including unacceptable inference time, imbalanced
training data, limited visual understanding, challenges with multi-turn
interactions, simplified language generation from robotic experiences, and
difficulties in handling on-the-fly unexpected situations like environmental
dynamics and task changes.

基于视频和语言模型的 DriVLMe 智能代理在自动驾驶中表现出相对竞争力，但仍存在推理时间过长、训练数据不平衡、视觉理解能力有限、多轮交互挑战以及处理环境动态和任务变化等实时场景中的困难。

DriVLMe: 用具身化和社交经验增强基于 LLM 的自动驾驶代理

DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and  Social Experiences

This paper studies the problem of image-goal navigation which involves
navigating to the location indicated by a goal image in a novel previously
unseen environment. To tackle this problem, we design topological
representations for space that effectively leverage semantics and afford
approximate geometric reasoning. At the heart of our representations are nodes
with associated semantic features, that are interconnected using coarse
geometric information. We describe supervised learning-based algorithms that
can build, maintain and use such representations under noisy actuation.
Experimental study in visually and physically realistic simulation suggests
that our method builds effective representations that capture structural
regularities and efficiently solve long-horizon navigation problems. We observe
a relative improvement of more than 50% over existing methods that study this
task.

本文研究图像目标导航问题，提出了一种基于拓扑表示法、语义特征和监督学习算法的解决方案，在视觉和物理仿真实验中部署该算法，与现有方法相比，在长时间任务中呈现出 50％以上的相对提高。