Large language models (LLMs) are accelerating the development of
language-guided robot planners. Meanwhile, symbolic planners offer the
advantage of interpretability. This paper proposes a new task that bridges
these two trends, namely, multimodal planning problem specification. The aim is
to generate a problem description (PD), a machine-readable file used by the
planners to find a plan. By generating PDs from language instruction and scene
observation, we can drive symbolic planners in a language-guided framework. We
propose a Vision-Language Interpreter (ViLaIn), a new framework that generates
PDs using state-of-the-art LLM and vision-language models. ViLaIn can refine
generated PDs via error message feedback from the symbolic planner. Our aim is
to answer the question: How accurately can ViLaIn and the symbolic planner
generate valid robot plans? To evaluate ViLaIn, we introduce a novel dataset
called the problem description generation (ProDG) dataset. The framework is
evaluated with four new evaluation metrics. Experimental results show that
ViLaIn can generate syntactically correct problems with more than 99% accuracy
and valid plans with more than 58% accuracy.

提出了一种将语言模型和符号规划器相结合的新框架，通过生成语言指令和场景观察的问题描述 (PD) 来驱动符号规划器，实现语言引导的机器人规划，实验结果显示该框架能够以超过 99% 的准确率生成句法正确的问题描述和超过 58% 的准确率生成有效的机器人规划。