Legged robots are physically capable of navigating a diverse variety of
environments and overcoming a wide range of obstructions. For example, in a
search and rescue mission, a legged robot could climb over debris, crawl
through gaps, and navigate out of dead ends. However, the robot's controller
needs to respond intelligently to such varied obstacles, and this requires
handling unexpected and unusual scenarios successfully. This presents an open
challenge to current learning methods, which often struggle with generalization
to the long tail of unexpected situations without heavy human supervision. To
address this issue, we investigate how to leverage the broad knowledge about
the structure of the world and commonsense reasoning capabilities of
vision-language models (VLMs) to aid legged robots in handling difficult,
ambiguous situations. We propose a system, VLM-Predictive Control (VLM-PC),
combining two key components that we find to be crucial for eliciting
on-the-fly, adaptive behavior selection with VLMs: (1) in-context adaptation
over previous robot interactions and (2) planning multiple skills into the
future and replanning. We evaluate VLM-PC on several challenging real-world
obstacle courses, involving dead ends and climbing and crawling, on a Go1
quadruped robot. Our experiments show that by reasoning over the history of
interactions and future plans, VLMs enable the robot to autonomously perceive,
navigate, and act in a wide range of complex scenarios that would otherwise
require environment-specific engineering or human guidance.

通过利用视觉 - 语言模型（VLM）的通识推理能力，以及 VLM 在上下文适应和多技能规划方面的关键组件，我们提出了 VLM-Predictive Control（VLM-PC）系统，能够帮助双足机器人在处理困难、模糊的情况下做出自适应行为选择，从而使机器人能够在多样复杂的情景中自主感知、导航和行动，而无需环境特定的工程或人类引导。