The financial domain frequently deals with large numbers of long documents
that are essential for daily operations. Significant effort is put towards
automating financial data analysis. However, a persistent challenge, not
limited to the finance domain, is the scarcity of datasets that accurately
reflect real-world tasks for model evaluation. Existing datasets are often
constrained by size, context, or relevance to practical applications. Moreover,
LLMs are currently trained on trillions of tokens of text, limiting access to
novel data or documents that models have not encountered during training for
unbiased evaluation. We propose SEC-QA, a continuous dataset generation
framework with two key features: 1) the semi-automatic generation of
Question-Answer (QA) pairs spanning multiple long context financial documents,
which better represent real-world financial scenarios; 2) the ability to
continually refresh the dataset using the most recent public document
collections, not yet ingested by LLMs. Our experiments show that current
retrieval augmented generation methods systematically fail to answer these
challenging multi-document questions. In response, we introduce a QA system
based on program-of-thought that improves the ability to perform complex
information retrieval and quantitative reasoning pipelines, thereby increasing
QA accuracy.

金融数据分析中普遍面临的挑战是找到能准确反映实际任务用于模型评估的数据集，作者提出了一种连续的数据集生成框架 SEC-QA，在这个框架中，提供了两个重要功能，分别是跨多个长文本金融文件生成问题 - 答案 (QA) 对，以更好地代表真实世界金融场景，并且能够利用最新的公共文件集刷新数据集，这些文件集 LLMs 尚未接触到。实验证明，当前的检索增强型生成方法不能够有效回答这些复杂的多文档问题，为此，作者引入了一种基于思维程序的 QA 系统，提高了复杂信息检索和定量推理流程的能力，从而提高了 QA 准确性。