Building a conversational embodied agent to execute real-life tasks has been
a long-standing yet quite challenging research goal, as it requires effective
human-agent communication, multi-modal understanding, long-range sequential
decision making, etc. Traditional symbolic methods have scaling and
generalization issues, while end-to-end deep learning models suffer from data
scarcity and high task complexity, and are often hard to explain. To benefit
from both worlds, we propose JARVIS, a neuro-symbolic commonsense reasoning
framework for modular, generalizable, and interpretable conversational embodied
agents. First, it acquires symbolic representations by prompting large language
models (LLMs) for language understanding and sub-goal planning, and by
constructing semantic maps from visual observations. Then the symbolic module
reasons for sub-goal planning and action generation based on task- and
action-level common sense. Extensive experiments on the TEACh dataset validate
the efficacy and efficiency of our JARVIS framework, which achieves
state-of-the-art (SOTA) results on all three dialog-based embodied tasks,
including Execution from Dialog History (EDH), Trajectory from Dialog (TfD),
and Two-Agent Task Completion (TATC) (e.g., our method boosts the unseen
Success Rate on EDH from 6.1\% to 15.8\%). Moreover, we systematically analyze
the essential factors that affect the task performance and also demonstrate the
superiority of our method in few-shot settings. Our JARVIS model ranks first in
the Alexa Prize SimBot Public Benchmark Challenge.

通过将大语言模型与视觉信息相结合，建立了一种神经符号共同思考推理框架 JARVIS，用于构建可解释且高效的对话体载体，结果达到了现有方法中最优结果。

JARVIS: 一种面向对话体现代理的神经符号常识推理框架

JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agents

Despite recent progress, learning new tasks through language instructions
remains an extremely challenging problem. On the ALFRED benchmark for task
learning, the published state-of-the-art system only achieves a task success
rate of less than 10% in an unseen environment, compared to the human
performance of over 90%. To address this issue, this paper takes a closer look
at task learning. In a departure from a widely applied end-to-end architecture,
we decomposed task learning into three sub-problems: sub-goal planning, scene
navigation, and object manipulation; and developed a model HiTUT (stands for
Hierarchical Tasks via Unified Transformers) that addresses each sub-problem in
a unified manner to learn a hierarchical task structure. On the ALFRED
benchmark, HiTUT has achieved the best performance with a remarkably higher
generalization ability. In the unseen environment, HiTUT achieves over 160%
performance gain in success rate compared to the previous state of the art. The
explicit representation of task structures also enables an in-depth
understanding of the nature of the problem and the ability of the agent, which
provides insight for future benchmark development and evaluation.

本文提出一个名称为 HiTUT 的模型，将任务学习分解成三个子问题：子目标规划、场景导航和物体操作，并统一解决了每个子问题，以学习分层任务结构。在 ALFRED 基准测试中，HiTUT 表现出最好的性能和更高的泛化能力。