We introduce SpreadsheetBench, a challenging spreadsheet manipulation
benchmark exclusively derived from real-world scenarios, designed to immerse
current large language models (LLMs) in the actual workflow of spreadsheet
users. Unlike existing benchmarks that rely on synthesized queries and
simplified spreadsheet files, SpreadsheetBench is built from 912 real questions
gathered from online Excel forums, which reflect the intricate needs of users.
The associated spreadsheets from the forums contain a variety of tabular data
such as multiple tables, non-standard relational tables, and abundant
non-textual elements. Furthermore, we propose a more reliable evaluation metric
akin to online judge platforms, where multiple spreadsheet files are created as
test cases for each instruction, ensuring the evaluation of robust solutions
capable of handling spreadsheets with varying values. Our comprehensive
evaluation of various LLMs under both single-round and multi-round inference
settings reveals a substantial gap between the state-of-the-art (SOTA) models
and human performance, highlighting the benchmark's difficulty.

通过从在线 Excel 论坛收集的 912 个真实问题和相关表格构建的 SpreadsheetBench 测试数据集，我们介绍了一种挑战性的电子表格处理基准，旨在使当前的大型语言模型真正融入电子表格用户的实际工作流程中。我们提出了一种更可靠的评估指标，并对不同的大型语言模型在单轮和多轮推断条件下进行综合评估，结果显示这个基准测试非常困难，并且是目前最先进模型和人类性能之间的差距。

电子表格测评：迈向挑战现实世界电子表格操作

SpreadsheetBench: Towards Challenging Real World Spreadsheet  Manipulation

Spreadsheet manipulation is widely existing in most daily works and
significantly improves working efficiency. Large language model (LLM) has been
recently attempted for automatic spreadsheet manipulation but has not yet been
investigated in complicated and realistic tasks where reasoning challenges
exist (e.g., long horizon manipulation with multi-step reasoning and ambiguous
requirements). To bridge the gap with the real-world requirements, we introduce
$\textbf{SheetRM}$, a benchmark featuring long-horizon and multi-category tasks
with reasoning-dependent manipulation caused by real-life challenges. To
mitigate the above challenges, we further propose $\textbf{SheetAgent}$, a
novel autonomous agent that utilizes the power of LLMs. SheetAgent consists of
three collaborative modules: $\textit{Planner}$, $\textit{Informer}$, and
$\textit{Retriever}$, achieving both advanced reasoning and accurate
manipulation over spreadsheets without human interaction through iterative task
reasoning and reflection. Extensive experiments demonstrate that SheetAgent
delivers 20-30% pass rate improvements on multiple benchmarks over baselines,
achieving enhanced precision in spreadsheet manipulation and demonstrating
superior table reasoning abilities. More details and visualizations are
available at this https URL

在复杂而真实的任务中，通过使用大规模语言模型（LLM）进一步提出 SheetAgent，一个能够推理和操作表格的自主代理模型，通过迭代任务推理和反思实现先进的推理和准确的电子表格操作，并在多个基准测试中实现了 20-30% 的通过率的提升和卓越的表格推理能力。