Humans can quickly learn new behaviors by leveraging background world
knowledge. In contrast, agents trained with reinforcement learning (RL)
typically learn behaviors from scratch. We thus propose a novel approach that
uses the vast amounts of general and indexable world knowledge encoded in
vision-language models (VLMs) pre-trained on Internet-scale data for embodied
RL. We initialize policies with VLMs by using them as promptable
representations: embeddings that are grounded in visual observations and encode
semantic features based on the VLM's internal knowledge, as elicited through
prompts that provide task context and auxiliary information. We evaluate our
approach on visually-complex, long horizon RL tasks in Minecraft and robot
navigation in Habitat. We find that our policies trained on embeddings
extracted from general-purpose VLMs outperform equivalent policies trained on
generic, non-promptable image embeddings. We also find our approach outperforms
instruction-following methods and performs comparably to domain-specific
embeddings.

通过利用背景世界知识，人类能够快速学习新的行为方式。相比之下，强化学习训练的代理通常需要从零开始学习行为。因此，我们提出了一种新的方法，利用基于视觉语言模型（VLMs）的通用世界知识和可索引知识，这些模型在互联网规模的数据上进行预训练，用于实体强化学习。我们通过将 VLMs 用作可提示的表示方式来初始化策略：通过提示提供任务背景和辅助信息，这些嵌入基于视觉观察进行了接地，并编码了 VLM 的内部知识的语义特征。我们在 Minecraft 中的视觉复杂、长期的强化学习任务以及 Habitat 中的机器人导航任务上评估了我们的方法。我们发现，与从通用的非可提示图像嵌入训练的等效策略相比，我们基于通用 VLMs 提取的嵌入的训练策略表现更好。我们还发现，我们的方法优于遵循指令的方法，并与特定领域的嵌入方法效果相当。