We introduce NATURAL PLAN, a realistic planning benchmark in natural language
containing 3 key tasks: Trip Planning, Meeting Planning, and Calendar
Scheduling. We focus our evaluation on the planning capabilities of LLMs with
full information on the task, by providing outputs from tools such as Google
Flights, Google Maps, and Google Calendar as contexts to the models. This
eliminates the need for a tool-use environment for evaluating LLMs on Planning.
We observe that NATURAL PLAN is a challenging benchmark for state of the art
models. For example, in Trip Planning, GPT-4 and Gemini 1.5 Pro could only
achieve 31.1% and 34.8% solve rate respectively. We find that model performance
drops drastically as the complexity of the problem increases: all models
perform below 5% when there are 10 cities, highlighting a significant gap in
planning in natural language for SoTA LLMs. We also conduct extensive ablation
studies on NATURAL PLAN to further shed light on the (in)effectiveness of
approaches such as self-correction, few-shot generalization, and in-context
planning with long-contexts on improving LLM planning.

NATURAL PLAN 是一个真实的自然语言规划基准，包含三个关键任务：旅行规划、会议规划和日历安排。在任务的全信息下评估了 LLMs 的规划能力，通过提供 Google Flights、Google Maps 和 Google Calendar 等工具的输出作为模型的上下文，消除了在规划评估中使用工具环境的需要。我们观察到 NATURAL PLAN 对于现有模型来说是一个具有挑战性的基准。例如，在旅行规划中，GPT-4 和 Gemini 1.5 Pro 仅能分别实现 31.1% 和 34.8% 的解决率。当问题复杂度增加时，模型性能急剧下降：当有 10 个城市时，所有模型的表现均低于 5%，凸显了现有 LLMs 在自然语言规划方面存在的重大差距。我们还对 NATURAL PLAN 进行了全面的剔除研究，以进一步阐明自我修正、少样本泛化以及长上下文中的规划对于改善 LLM 规划的 (无) 效性。