In this paper, we present an exploration of LLMs' abilities to problem solve
with physical reasoning in situated environments. We construct a simple
simulated environment and demonstrate examples of where, in a zero-shot
setting, both text and multimodal LLMs display atomic world knowledge about
various objects but fail to compose this knowledge in correct solutions for an
object manipulation and placement task. We also use BLIP, a vision-language
model trained with more sophisticated cross-modal attention, to identify cases
relevant to object physical properties that that model fails to ground.
Finally, we present a procedure for discovering the relevant properties of
objects in the environment and propose a method to distill this knowledge back
into the LLM.

本文探讨了 LLMs 在情境环境中通过物理推理解决问题的能力。我们构建了一个简单的模拟环境，并展示了 LLMs 在无先验条件下对各种物体的原子世界知识的示例，但在对象操作和放置任务中未能将这些知识组合成正确的解决方案。我们还使用了 BLIP，一个经过更复杂的跨模态注意力训练的视觉语言模型，来识别该模型未能基于对象物理属性进行建模的案例。最后，我们提出了一种发现环境中相关性质的对象的方法，并提出了一种将这些知识转化回 LLM 的方法。

探索基于多模态推理的物理动力学失败案例

Exploring Failure Cases in Multimodal Reasoning About Physical Dynamics

Task planning can require defining myriad domain knowledge about the world in
which a robot needs to act. To ameliorate that effort, large language models
(LLMs) can be used to score potential next actions during task planning, and
even generate action sequences directly, given an instruction in natural
language with no additional domain information. However, such methods either
require enumerating all possible next steps for scoring, or generate free-form
text that may contain actions not possible on a given robot in its current
context. We present a programmatic LLM prompt structure that enables plan
generation functional across situated environments, robot capabilities, and
tasks. Our key insight is to prompt the LLM with program-like specifications of
the available actions and objects in an environment, as well as with example
programs that can be executed. We make concrete recommendations about prompt
structure and generation constraints through ablation experiments, demonstrate
state of the art success rates in VirtualHome household tasks, and deploy our
method on a physical robot arm for tabletop tasks. Website at
progprompt.github.io

通过使用编程式的提示结构，可以促使大型语言模型在不同的环境、机器人能力和任务中进行计划生成，从而大大减少了任务规划中需要定义的领域知识。

使用大型语言模型生成机器人任务计划

ProgPrompt: Generating Situated Robot Task Plans using Large Language  Models

Intelligent agents that are confronted with novel concepts in situated
environments will need to ask their human teammates questions to learn about
the physical world. To better understand this problem, we need data about
asking questions in situated task-based interactions. To this end, we present
the Human-Robot Dialogue Learning (HuRDL) Corpus - a novel dialogue corpus
collected in an online interactive virtual environment in which human
participants play the role of a robot performing a collaborative
tool-organization task. We describe the corpus data and a corresponding
annotation scheme to offer insight into the form and content of questions that
humans ask to facilitate learning in a situated environment. We provide the
corpus as an empirically-grounded resource for improving question generation in
situated intelligent agents.

通过收集 Human-Robot Dialogue Learning (HuRDL) Corpus 数据集，提出利用人类角色扮演机器人在在线交互虚拟环境中进行协作任务，描述了对话数据和对应注释方案，为智能代理的提高提出了帮助学习的问题。