How-to procedures, such as how to plant a garden, are ubiquitous. But one
size does not fit all - humans often need to customize these procedural plans
according to their specific needs, e.g., planting a garden without pesticides.
While LLMs can fluently generate generic procedures, we present the first study
on how well LLMs can customize open-domain procedures. We introduce
CustomPlans, a probe dataset of customization hints that encodes diverse user
needs for open-domain How-to procedures. Using LLMs as CustomizationAgent and
ExecutionAgent in different settings, we establish their abilities to perform
open-domain procedure customization. Human evaluation shows that using these
agents in a Sequential setting is the best, but they are good enough only ~51%
of the time. Error analysis shows that LLMs do not sufficiently address user
customization needs in their generated procedures.

首次研究了 LLMs 能够多大程度上个性化定制开放领域的操作过程，使用 CustomPlans 数据集作为个性化代理和执行代理，人工评估结果显示在顺序设置中使用这些代理效果最佳，但仅有约 51% 的准确率，误差分析显示 LLMs 在生成的过程中不能充分满足用户的个性化需求。