In this paper, we argue that the next generation of robots can be commanded
using only Language Models' prompts. Every prompt interrogates separately a
specific Robotic Modality via its Modality Language Model (MLM). A central Task
Modality mediates the whole communication to execute the robotic mission via a
Large Language Model (LLM). This paper gives this new robotic design pattern
the name of: Prompting Robotic Modalities (PRM). Moreover, this paper applies
this PRM design pattern in building a new robotic framework named
ROSGPT_Vision. ROSGPT_Vision allows the execution of a robotic task using only
two prompts: a Visual and an LLM prompt. The Visual Prompt extracts, in natural
language, the visual semantic features related to the task under consideration
(Visual Robotic Modality). Meanwhile, the LLM Prompt regulates the robotic
reaction to the visual description (Task Modality). The framework automates all
the mechanisms behind these two prompts. The framework enables the robot to
address complex real-world scenarios by processing visual data, making informed
decisions, and carrying out actions automatically. The framework comprises one
generic vision module and two independent ROS nodes. As a test application, we
used ROSGPT_Vision to develop CarMate, which monitors the driver's distraction
on the roads and makes real-time vocal notifications to the driver. We showed
how ROSGPT_Vision significantly reduced the development cost compared to
traditional methods. We demonstrated how to improve the quality of the
application by optimizing the prompting strategies, without delving into
technical details. ROSGPT_Vision is shared with the community (link:
this https URL) to advance robotic research in this
direction and to build more robotic frameworks that implement the PRM design
pattern and enables controlling robots using only prompts.

通过使用语言模型提示，我们提出了新一代机器人可以通过使用机器人模态语言模型（MLM）单独独立地与特定的机器人模态进行通信。一个中央任务模态通过大型语言模型（LLM）中介整个通信过程以执行机器人任务。我们将这种新的机器人设计模式命名为 Prompting Robotic Modalities（PRM），并在一个新的机器人框架 ROSGPT_Vision 中应用了这种 PRM 设计模式。ROSGPT_Vision 通过仅使用两个提示（一个视觉提示和一个 LLM 提示）来执行机器人任务，实现了处理视觉数据、做出明智决策和自动执行操作的能力。