LLM-based agents have emerged as promising tools, which are crafted to
fulfill complex tasks by iterative planning and action. However, these agents
are susceptible to undesired planning hallucinations when lacking specific
knowledge for expertise-intensive tasks. To address this, preliminary attempts
are made to enhance planning reliability by incorporating external
workflow-related knowledge. Despite the promise, such infused knowledge is
mostly disorganized and diverse in formats, lacking rigorous formalization and
comprehensive comparisons. Motivated by this, we formalize different formats of
workflow knowledge and present FlowBench, the first benchmark for
workflow-guided planning. FlowBench covers 51 different scenarios from 6
domains, with knowledge presented in diverse formats. To assess different LLMs
on FlowBench, we design a multi-tiered evaluation framework. We evaluate the
efficacy of workflow knowledge across multiple formats, and the results
indicate that current LLM agents need considerable improvements for
satisfactory planning. We hope that our challenging benchmark can pave the way
for future agent planning research.

LLM 基于智能体是有前途的工具，用于通过迭代规划和行动来完成复杂任务。然而，在缺乏专业知识进行任务时，这些智能体容易出现计划幻觉。为了解决这个问题，初步尝试通过整合外部与工作流相关的知识来提高规划的可靠性。我们形式化了不同格式的工作流知识，并提出了 FlowBench，这是用于工作流指导规划的第一个基准。FlowBench 涵盖了 6 个领域中的 51 个不同场景，并以多种格式呈现知识。为了评估 LLM 在 FlowBench 上的表现，我们设计了一个多层次评估框架。我们评估了不同格式的工作流知识的功效，结果表明目前的 LLM 智能体在满意的规划方面需要相当的改进。我们希望我们具有挑战性的基准可以为未来的智能体规划研究铺平道路。