Recent advancements on Large Language Models (LLMs) enable AI Agents to
automatically generate and execute multi-step plans to solve complex tasks.
However, since LLM's content generation process is hardly controllable, current
LLM-based agents frequently generate invalid or non-executable plans, which
jeopardizes the performance of the generated plans and corrupts users' trust in
LLM-based agents. In response, this paper proposes a novel ``Formal-LLM''
framework for LLM-based agents by integrating the expressiveness of natural
language and the precision of formal language. Specifically, the framework
allows human users to express their requirements or constraints for the
planning process as an automaton. A stack-based LLM plan generation process is
then conducted under the supervision of the automaton to ensure that the
generated plan satisfies the constraints, making the planning process
controllable. We conduct experiments on both benchmark tasks and practical
real-life tasks, and our framework achieves over 50% overall performance
increase, which validates the feasibility and effectiveness of employing
Formal-LLM to guide the plan generation of agents, preventing the agents from
generating invalid and unsuccessful plans. Further, more controllable LLM-based
agents can facilitate the broader utilization of LLM in application scenarios
where high validity of planning is essential. The work is open-sourced at
this https URL

该篇论文提出了一种名为 ``Formal-LLM'' 的新型框架用于基于 LLM 的代理，通过结合自然语言的表现力和形式语言的准确性，使规划过程具有控制性，从而防止代理生成无效和不成功的计划，并且通过实验验证了该框架在任务性能上取得的超过 50% 的整体性能提升，为在规划的有效性至关重要的应用场景中更广泛地利用 LLM 提供了可能性。