As large language models (LLMs) impact a growing number of complex domains,
it is becoming increasingly important to have fair, accurate, and rigorous
evaluation benchmarks. Evaluating the reasoning skills required for business
and financial NLP stands out as a particularly difficult challenge. We
introduce BizBench, a new benchmark for evaluating models' ability to reason
about realistic financial problems. BizBench comprises 8 quantitative reasoning
tasks. Notably, BizBench targets the complex task of question-answering (QA)
for structured and unstructured financial data via program synthesis (i.e.,
code generation). We introduce three diverse financially-themed code-generation
tasks from newly collected and augmented QA data. Additionally, we isolate
distinct financial reasoning capabilities required to solve these QA tasks:
reading comprehension of financial text and tables, which is required to
extract correct intermediate values; and understanding domain knowledge (e.g.,
financial formulas) needed to calculate complex solutions. Collectively, these
tasks evaluate a model's financial background knowledge, ability to extract
numeric entities from financial documents, and capacity to solve problems with
code. We conduct an in-depth evaluation of open-source and commercial LLMs,
illustrating that BizBench is a challenging benchmark for quantitative
reasoning in the finance and business domain.

评估金融推理能力的基准和挑战性问题 BizBench，通过三个财务领域的代码生成任务对大型语言模型进行了全面深入的评估，并表明 BizBench 是金融和商业领域量化推理的一个具有挑战性的基准模型。

BizBench：商业和金融的定量推理基准

BizBench: A Quantitative Reasoning Benchmark for Business and Finance

Large Language Models (LLMs), consisting of 100 billion or more parameters,
have demonstrated remarkable ability in complex multi-step reasoning tasks.
However, the application of such generic advancements has been limited to a few
fields, such as clinical or legal, with the field of financial reasoning
remaining largely unexplored. To the best of our knowledge, the ability of LLMs
to solve financial reasoning problems has never been dealt with, and whether it
can be performed at any scale remains unknown. To address this knowledge gap,
this research presents a comprehensive investigation into the potential
application of LLMs in the financial domain. The investigation includes a
detailed exploration of a range of subjects, including task formulation,
synthetic data generation, prompting methods, and evaluation capability.
Furthermore, the study benchmarks various GPT variants with parameter scales
ranging from 2.8B to 13B, with and without instruction tuning, on diverse
dataset sizes. By analyzing the results, we reveal that the ability to generate
coherent financial reasoning first emerges at 6B parameters, and continues to
improve with better instruction-tuning or larger datasets. Additionally, the
study provides a publicly accessible dataset named sFIOG (Synthetic-Financial
Investment Opinion Generation), consisting of 11,802 synthetic investment
thesis samples, to support further research in the field of financial
reasoning. Overall, this research seeks to contribute to the understanding of
the efficacy of language models in the field of finance, with a particular
emphasis on their ability to engage in sophisticated reasoning and analysis
within the context of investment decision-making.

本文对大规模语言模型在金融领域中的应用进行了可行性研究，探讨了任务形式、数据生成、提示方法和评估能力等方面，发现在 6B 参数时可以生成相关的金融推理结果，并提供了名为 sFIOG 的公开数据集，以支持未来的研究。