As large language models (LLMs) impact a growing number of complex domains,
it is becoming increasingly important to have fair, accurate, and rigorous
evaluation benchmarks. Evaluating the reasoning skills required for business
and financial NLP stands out as a particularly difficult challenge. We
introduce BizBench, a new benchmark for evaluating models' ability to reason
about realistic financial problems. BizBench comprises 8 quantitative reasoning
tasks. Notably, BizBench targets the complex task of question-answering (QA)
for structured and unstructured financial data via program synthesis (i.e.,
code generation). We introduce three diverse financially-themed code-generation
tasks from newly collected and augmented QA data. Additionally, we isolate
distinct financial reasoning capabilities required to solve these QA tasks:
reading comprehension of financial text and tables, which is required to
extract correct intermediate values; and understanding domain knowledge (e.g.,
financial formulas) needed to calculate complex solutions. Collectively, these
tasks evaluate a model's financial background knowledge, ability to extract
numeric entities from financial documents, and capacity to solve problems with
code. We conduct an in-depth evaluation of open-source and commercial LLMs,
illustrating that BizBench is a challenging benchmark for quantitative
reasoning in the finance and business domain.

评估金融推理能力的基准和挑战性问题 BizBench，通过三个财务领域的代码生成任务对大型语言模型进行了全面深入的评估，并表明 BizBench 是金融和商业领域量化推理的一个具有挑战性的基准模型。