We introduce Voyager, the first LLM-powered embodied lifelong learning agent
in Minecraft that continuously explores the world, acquires diverse skills, and
makes novel discoveries without human intervention. Voyager consists of three
key components: 1) an automatic curriculum that maximizes exploration, 2) an
ever-growing skill library of executable code for storing and retrieving
complex behaviors, and 3) a new iterative prompting mechanism that incorporates
environment feedback, execution errors, and self-verification for program
improvement. Voyager interacts with GPT-4 via blackbox queries, which bypasses
the need for model parameter fine-tuning. The skills developed by Voyager are
temporally extended, interpretable, and compositional, which compounds the
agent's abilities rapidly and alleviates catastrophic forgetting. Empirically,
Voyager shows strong in-context lifelong learning capability and exhibits
exceptional proficiency in playing Minecraft. It obtains 3.3x more unique
items, travels 2.3x longer distances, and unlocks key tech tree milestones up
to 15.3x faster than prior SOTA. Voyager is able to utilize the learned skill
library in a new Minecraft world to solve novel tasks from scratch, while other
techniques struggle to generalize. We open-source our full codebase and prompts
at this https URL

论文介绍了 Voyager：基于 LLM 的具身化终身学习代理，其由自动课程、可执行代码的技能库和新的迭代提示机制组成，可以在 Minecraft 中探索世界、获得多种技能和进行创新发现。在实证研究中，Voyager 表现出强大的上下文终身学习能力，在 Minecraft 游戏中比其他技术更快地解锁关键技术树里程碑，开发者在 https URL 上公开了代码库和提示。

Voyager: 具备大型语言模型的开放式智能体

Voyager: An Open-Ended Embodied Agent with Large Language Models

Many real-world problems are compositional - solving them requires completing
interdependent sub-tasks, either in series or in parallel, that can be
represented as a dependency graph. Deep reinforcement learning (RL) agents
often struggle to learn such complex tasks due to the long time horizons and
sparse rewards. To address this problem, we present Compositional Design of
Environments (CoDE), which trains a Generator agent to automatically build a
series of compositional tasks tailored to the RL agent's current skill level.
This automatic curriculum not only enables the agent to learn more complex
tasks than it could have otherwise, but also selects tasks where the agent's
performance is weak, enhancing its robustness and ability to generalize
zero-shot to unseen tasks at test-time. We analyze why current environment
generation techniques are insufficient for the problem of generating
compositional tasks, and propose a new algorithm that addresses these issues.
Our results assess learning and generalization across multiple compositional
tasks, including the real-world problem of learning to navigate and interact
with web pages. We learn to generate environments composed of multiple pages or
rooms, and train RL agents capable of completing wide-range of complex tasks in
those environments. We contribute two new benchmark frameworks for generating
compositional tasks, compositional MiniGrid and gMiniWoB for web
navigation.CoDE yields 4x higher success rate than the strongest baseline, and
demonstrates strong performance of real websites learned on 3500 primitive
tasks.

通过训练生成器创建任务序列的自动课程表，以使强化学习代理能够完成复杂的组合任务，提高其健壮性和泛化能力，我们提出了一种名为 CoDE 的新算法，并对其进行了分析和评估

零样本组合强化学习环境生成

Environment Generation for Zero-Shot Compositional Reinforcement Learning

Continually solving new, unsolved tasks is the key to learning diverse
behaviors. Through reinforcement learning (RL), we have made massive strides
towards solving tasks that have a single goal. However, in the multi-task
domain, where an agent needs to reach multiple goals, the choice of training
goals can largely affect sample efficiency. When biological agents learn, there
is often an organized and meaningful order to which learning happens. Inspired
by this, we propose setting up an automatic curriculum for goals that the agent
needs to solve. Our key insight is that if we can sample goals at the frontier
of the set of goals that an agent is able to reach, it will provide a
significantly stronger learning signal compared to randomly sampled goals. To
operationalize this idea, we introduce a goal proposal module that prioritizes
goals that maximize the epistemic uncertainty of the Q-function of the policy.
This simple technique samples goals that are neither too hard nor too easy for
the agent to solve, hence enabling continual improvement. We evaluate our
method across 13 multi-goal robotic tasks and 5 navigation tasks, and
demonstrate performance gains over current state-of-the-art methods.

通过提出自动课程设计和目标建议模块，来提高强化学习中的多任务目标采样效率，并在机器人和导航任务中展示了比现有方法更好的性能。