Reinforcement learning algorithms typically struggle in the absence of a
dense, well-shaped reward function. Intrinsically motivated exploration methods
address this limitation by rewarding agents for visiting novel states or
transitions, but these methods offer limited benefits in large environments
where most discovered novelty is irrelevant for downstream tasks. We describe a
method that uses background knowledge from text corpora to shape exploration.
This method, called ELLM (Exploring with LLMs) rewards an agent for achieving
goals suggested by a language model prompted with a description of the agent's
current state. By leveraging large-scale language model pretraining, ELLM
guides agents toward human-meaningful and plausibly useful behaviors without
requiring a human in the loop. We evaluate ELLM in the Crafter game environment
and the Housekeep robotic simulator, showing that ELLM-trained agents have
better coverage of common-sense behaviors during pretraining and usually match
or improve performance on a range of downstream tasks.

提出了一种名为 ELLM（LLM 探索）的方法，它利用来自文本语料库的背景知识来塑造探索，通过利用大规模语言模型预训练，无需人介入便能引导智能体朝向有人类意义的和可能有用的行为方向，通过在 Crafter 游戏环境和 Housekeep 机器人模拟器中的实验，证明了 ELLM 训练的代理在预训练期间具有更好的常识行为覆盖，并且在一系列下游任务中通常与或优于性能。