Planning is an important capability of artificial agents that perform
long-horizon tasks in real-world environments. In this work, we explore the use
of pre-trained language models (PLMs) to reason about plan sequences from text
instructions in embodied visual environments. Prior PLM based approaches for
planning either assume observations are available in the form of text (e.g.,
provided by a captioning model), reason about plans from the instruction alone,
or incorporate information about the visual environment in limited ways (such
as a pre-trained affordance function). In contrast, we show that PLMs can
accurately plan even when observations are directly encoded as input prompts
for the PLM. We show that this simple approach outperforms prior approaches in
experiments on the ALFWorld and VirtualHome benchmarks.

本文研究利用预训练的语言模型（PLMs）从文本指令中推理出在具体视觉环境下的计划序列，结果表明 PLMs 能够准确地规划，且强于其他方法。

图像胜千言：语言模型基于像素进行规划

A Picture is Worth a Thousand Words: Language Models Plan from Pixels

Given a simple request like Put a washed apple in the kitchen fridge, humans
can reason in purely abstract terms by imagining action sequences and scoring
their likelihood of success, prototypicality, and efficiency, all without
moving a muscle. Once we see the kitchen in question, we can update our
abstract plans to fit the scene. Embodied agents require the same abilities,
but existing work does not yet provide the infrastructure necessary for both
reasoning abstractly and executing concretely. We address this limitation by
introducing ALFWorld, a simulator that enables agents to learn abstract, text
based policies in TextWorld (C\^ot\'e et al., 2018) and then execute goals from
the ALFRED benchmark (Shridhar et al., 2020) in a rich visual environment.
ALFWorld enables the creation of a new BUTLER agent whose abstract knowledge,
learned in TextWorld, corresponds directly to concrete, visually grounded
actions. In turn, as we demonstrate empirically, this fosters better agent
generalization than training only in the visually grounded environment.
BUTLER's simple, modular design factors the problem to allow researchers to
focus on models for improving every piece of the pipeline (language
understanding, planning, navigation, and visual scene understanding).

ALFWorld 介绍了一个新的模拟器，使得具有身体能力的代理人能够在 TextWorld 中学习抽象的文本策略，然后在富有视觉感的环境中执行 ALFRED 基准测试的目标，以促进代理人的更好泛化。