Accurate evaluation of financial question answering (QA) systems necessitates
a comprehensive dataset encompassing diverse question types and contexts.
However, current financial QA datasets lack scope diversity and question
complexity. This work introduces FinTextQA, a novel dataset for long-form
question answering (LFQA) in finance. FinTextQA comprises 1,262 high-quality,
source-attributed QA pairs extracted and selected from finance textbooks and
government agency websites.Moreover, we developed a Retrieval-Augmented
Generation (RAG)-based LFQA system, comprising an embedder, retriever,
reranker, and generator. A multi-faceted evaluation approach, including human
ranking, automatic metrics, and GPT-4 scoring, was employed to benchmark the
performance of different LFQA system configurations under heightened noisy
conditions. The results indicate that: (1) Among all compared generators,
Baichuan2-7B competes closely with GPT-3.5-turbo in accuracy score; (2) The
most effective system configuration on our dataset involved setting the
embedder, retriever, reranker, and generator as Ada2, Automated Merged
Retrieval, Bge-Reranker-Base, and Baichuan2-7B, respectively; (3) models are
less susceptible to noise after the length of contexts reaching a specific
threshold.

该研究介绍了 FinTextQA，这是一个用于金融领域长篇问答的创新数据集，并开发了一个基于 RAG 的 LFQA 系统，通过多角度评估方法得出了在噪声环境下不同配置的 LFQA 系统的性能表现和模型对噪声的容忍度。

FinTextQA：长文本金融问答数据集

FinTextQA: A Dataset for Long-form Financial Question Answering

Financial analysis is an important tool for evaluating company performance.
Practitioners work to answer financial questions to make profitable investment
decisions, and use advanced quantitative analyses to do so. As a result,
Financial Question Answering (QA) is a question answering task that requires
deep reasoning about numbers. Furthermore, it is unknown how well pre-trained
language models can reason in the financial domain. The current
state-of-the-art requires a retriever to collect relevant facts about the
financial question from the text and a generator to produce a valid financial
program and a final answer. However, recently large language models like GPT-3
have achieved state-of-the-art performance on wide variety of tasks with just a
few shot examples. We run several experiments with GPT-3 and find that a
separate retrieval model and logic engine continue to be essential components
to achieving SOTA performance in this task, particularly due to the precise
nature of financial questions and the complex information stored in financial
documents. With this understanding, our refined prompt-engineering approach on
GPT-3 achieves near SOTA accuracy without any fine-tuning.

金融分析是评估公司绩效的重要工具，而金融问题回答是需要对数字进行深入推理的问答任务。最近的研究发现，使用大型语言模型如 GPT-3 进行财务领域的推理仍需要相关事实的检索模型和逻辑引擎，特别是由于金融问题的精确性和金融文档中存储的复杂信息。在此基础上，我们在 GPT-3 上采用改进的提示工程方法，实现了接近最先进的准确性，而无需进行任何精调。