Existing ML benchmarks lack the depth and diversity of annotations needed for
evaluating models on business process management (BPM) tasks. BPM is the
practice of documenting, measuring, improving, and automating enterprise
workflows. However, research has focused almost exclusively on one task - full
end-to-end automation using agents based on multimodal foundation models (FMs)
like GPT-4. This focus on automation ignores the reality of how most BPM tools
are applied today - simply documenting the relevant workflow takes 60% of the
time of the typical process optimization project. To address this gap we
present WONDERBREAD, the first benchmark for evaluating multimodal FMs on BPM
tasks beyond automation. Our contributions are: (1) a dataset containing 2928
documented workflow demonstrations; (2) 6 novel BPM tasks sourced from
real-world applications ranging from workflow documentation to knowledge
transfer to process improvement; and (3) an automated evaluation harness. Our
benchmark shows that while state-of-the-art FMs can automatically generate
documentation (e.g. recalling 88% of the steps taken in a video demonstration
of a workflow), they struggle to re-apply that knowledge towards finer-grained
validation of workflow completion (F1 < 0.3). We hope WONDERBREAD encourages
the development of more "human-centered" AI tooling for enterprise applications
and furthers the exploration of multimodal FMs for the broader universe of BPM
tasks. We publish our dataset and experiments here:
this https URL

为了解决现有机器学习基准缺乏用于评估企业流程管理（BPM）任务的深度和多样性的问题，研究人员提出了 WONDERBREAD，这是评估多模态基础模型在超出自动化的 BPM 任务上的第一个基准。他们的研究发现，目前最先进的基础模型可以自动生成文档，但在验证工作流完成度方面存在困难。通过这项研究，他们希望鼓励开发更多 “以人为中心” 的企业应用人工智能工具，并进一步探索多模态基础模型在更广泛的 BPM 任务中的应用。