Large Language Models (LLMs) trained using massive text datasets have
recently shown promise in generating action plans for robotic agents from high
level text queries. However, these models typically do not consider the robot's
environment, resulting in generated plans that may not actually be executable
due to ambiguities in the planned actions or environmental constraints. In this
paper, we propose an approach to generate environmentally-aware action plans
that can be directly mapped to executable agent actions. Our approach involves
integrating environmental objects and object relations as additional inputs
into LLM action plan generation to provide the system with an awareness of its
surroundings, resulting in plans where each generated action is mapped to
objects present in the scene. We also design a novel scoring function that,
along with generating the action steps and associating them with objects, helps
the system disambiguate among object instances and take into account their
states. We evaluate our approach using the VirtualHome simulator and the
ActivityPrograms knowledge base. Our results show that the action plans
generated from our system outperform prior work in terms of their correctness
and executability by 5.3% and 8.9% respectively.

本文提出了一种整合环境对象和对象关系作为额外输入的方法，从而产生与场景中的对象相映射的行动计划，并设计了一种新的评分函数，该方法在 VirtualHome 模拟器和 ActivityPrograms 知识库中取得了可靠的结果，其行动计划的正确性和可执行性分别优于之前的研究约 5.3％和 8.9％。