People often give instructions whose meaning is ambiguous without further
context, expecting that their actions or goals will disambiguate their
intentions. How can we build assistive agents that follow such instructions in
a flexible, context-sensitive manner? This paper introduces cooperative
language-guided inverse plan search (CLIPS), a Bayesian agent architecture for
pragmatic instruction following and goal assistance. Our agent assists a human
by modeling them as a cooperative planner who communicates joint plans to the
assistant, then performs multimodal Bayesian inference over the human's goal
from actions and language, using large language models (LLMs) to evaluate the
likelihood of an instruction given a hypothesized plan. Given this posterior,
our assistant acts to minimize expected goal achievement cost, enabling it to
pragmatically follow ambiguous instructions and provide effective assistance
even when uncertain about the goal. We evaluate these capabilities in two
cooperative planning domains (Doors, Keys & Gems and VirtualHome), finding that
CLIPS significantly outperforms GPT-4V, LLM-based literal instruction following
and unimodal inverse planning in both accuracy and helpfulness, while closely
matching the inferences and assistive judgments provided by human raters.

该研究介绍了合作语言引导的逆向计划搜索 (CLIPS)，这是一种贝叶斯代理架构，用于灵活、上下文敏感地遵循指令，提供目标协助。通过模型化人类为合作规划者，对动作和语言进行多模态贝叶斯推理，使用大型语言模型 (LLMs) 评估指令在假设计划下的可能性，辅助代理最小化预期目标达成成本，即使对目标存在不确定性，也能实现对模棱两可指令的实用跟踪和有效协助。在两个合作规划领域 (Doors, Keys & Gems 和 VirtualHome) 中评估了这些能力，发现 CLIPS 在准确性和帮助性方面明显优于 GPT-4V、基于 LLM 的字面指令跟踪和单模态逆向计划，并与人类评分者提供的推理和协助判断非常接近。