Large language models (LLMs) have recently received considerable attention as
alternative solutions for task planning. However, comparing the performance of
language-oriented task planners becomes difficult, and there exists a dearth of
detailed exploration regarding the effects of various factors such as
pre-trained model selection and prompt construction. To address this, we
propose a benchmark system for automatically quantifying performance of task
planning for home-service embodied agents. Task planners are tested on two
pairs of datasets and simulators: 1) ALFRED and AI2-THOR, 2) an extension of
Watch-And-Help and VirtualHome. Using the proposed benchmark system, we perform
extensive experiments with LLMs and prompts, and explore several enhancements
of the baseline planner. We expect that the proposed benchmark tool would
accelerate the development of language-oriented task planners.

提出了一个针对家庭服务实体代理的任务规划性能自动量化基准系统，对大型语言模型和提示进行了广泛实验，并探索了基线任务规划器的多个改进，预计该基准工具将加快语言导向的任务规划器的发展。