The financial domain frequently deals with large numbers of long documents
that are essential for daily operations. Significant effort is put towards
automating financial data analysis. However, a persistent challenge, not
limited to the finance domain, is the scarcity of datasets that accurately
reflect real-world tasks for model evaluation. Existing datasets are often
constrained by size, context, or relevance to practical applications. Moreover,
LLMs are currently trained on trillions of tokens of text, limiting access to
novel data or documents that models have not encountered during training for
unbiased evaluation. We propose SEC-QA, a continuous dataset generation
framework with two key features: 1) the semi-automatic generation of
Question-Answer (QA) pairs spanning multiple long context financial documents,
which better represent real-world financial scenarios; 2) the ability to
continually refresh the dataset using the most recent public document
collections, not yet ingested by LLMs. Our experiments show that current
retrieval augmented generation methods systematically fail to answer these
challenging multi-document questions. In response, we introduce a QA system
based on program-of-thought that improves the ability to perform complex
information retrieval and quantitative reasoning pipelines, thereby increasing
QA accuracy.

金融数据分析中普遍面临的挑战是找到能准确反映实际任务用于模型评估的数据集，作者提出了一种连续的数据集生成框架 SEC-QA，在这个框架中，提供了两个重要功能，分别是跨多个长文本金融文件生成问题 - 答案 (QA) 对，以更好地代表真实世界金融场景，并且能够利用最新的公共文件集刷新数据集，这些文件集 LLMs 尚未接触到。实验证明，当前的检索增强型生成方法不能够有效回答这些复杂的多文档问题，为此，作者引入了一种基于思维程序的 QA 系统，提高了复杂信息检索和定量推理流程的能力，从而提高了 QA 准确性。

SEC-QA: 金融问答的系统评估语料库

SEC-QA: A Systematic Evaluation Corpus for Financial QA

The sheer volume of financial statements makes it difficult for humans to
access and analyze a business's financials. Robust numerical reasoning likewise
faces unique challenges in this domain. In this work, we focus on answering
deep questions over financial data, aiming to automate the analysis of a large
corpus of financial documents. In contrast to existing tasks on general domain,
the finance domain includes complex numerical reasoning and understanding of
heterogeneous representations. To facilitate analytical progress, we propose a
new large-scale dataset, FinQA, with Question-Answering pairs over Financial
reports, written by financial experts. We also annotate the gold reasoning
programs to ensure full explainability. We further introduce baselines and
conduct comprehensive experiments in our dataset. The results demonstrate that
popular, large, pre-trained models fall far short of expert humans in acquiring
finance knowledge and in complex multi-step numerical reasoning on that
knowledge. Our dataset -- the first of its kind -- should therefore enable
significant, new community research into complex application domains. The
dataset and code are publicly availablehttps://github.com/czyssrs/FinQA.

为了解决金融领域中财务报表分析的难题，本文提出了一个新的数据集 FinQA，并进行了广泛的实验。实验结果显示预训练模型在金融知识和复杂多步骤的数值推理方面远远不及专家人类。这个新的数据集应该为复杂应用领域的深入研究提供了可能性。