This paper explores leveraging large language models for map-free off-road
navigation using generative AI, reducing the need for traditional data
collection and annotation. We propose a method where a robot receives verbal
instructions, converted to text through Whisper, and a large language model
(LLM) model extracts landmarks, preferred terrains, and crucial adverbs
translated into speed settings for constrained navigation. A language-driven
semantic segmentation model generates text-based masks for identifying
landmarks and terrain types in images. By translating 2D image points to the
vehicle's motion plane using camera parameters, an MPC controller can guides
the vehicle towards the desired terrain. This approach enhances adaptation to
diverse environments and facilitates the use of high-level instructions for
navigating complex and challenging terrains.

通过利用大型语言模型，该论文研究了使用生成 AI 进行无地图越野导航，从而减少对传统数据收集和注释的需求。通过将口头指令转换为文本，并使用大型语言模型提取地标、优选地形和关键副词来进行限制性导航的速度设置。通过语言驱动的语义分割模型生成基于文本的图像遮罩，以识别图像中的地标和地形类型。通过使用相机参数将 2D 图像点转换为车辆的运动平面，MPC 控制器可将车辆引导向所需地形。这种方法增强了对不同环境的适应能力，并便于使用高级指令导航复杂和具有挑战性的地形。

利用 LLMs 和语音指令在优先地形上进行约束的机器人导航：发挥副词的力量

Constrained Robotic Navigation on Preferred Terrains Using LLMs and  Speech Instruction: Exploiting the Power of Adverbs

This paper presents a research platform that supports spoken dialogue
interaction with multiple robots. The demonstration showcases our crafted
MultiBot testing scenario in which users can verbally issue search, navigate,
and follow instructions to two robotic teammates: a simulated ground robot and
an aerial robot. This flexible language and robotic platform takes advantage of
existing tools for speech recognition and dialogue management that are
compatible with new domains, and implements an inter-agent communication
protocol (tactical behavior specification), where verbal instructions are
encoded for tasks assigned to the appropriate robot.

该研究提供一种支持与多个机器人的口语对话交互的研究平台，演示了一个多机器人测试场景，用户可以用语音口令发出搜索、导航和跟随指令给两个机器人队友：一个模拟的地面机器人和一个空中机器人。该灵活的语言和机器人平台利用现有的语音识别和对话管理工具，并实现了一种智能体之间的通信协议（战术行为规范），在该协议中，口头指令被编码为分配给合适机器人的任务。