Accurate evaluation of financial question answering (QA) systems necessitates
a comprehensive dataset encompassing diverse question types and contexts.
However, current financial QA datasets lack scope diversity and question
complexity. This work introduces FinTextQA, a novel dataset for long-form
question answering (LFQA) in finance. FinTextQA comprises 1,262 high-quality,
source-attributed QA pairs extracted and selected from finance textbooks and
government agency websites.Moreover, we developed a Retrieval-Augmented
Generation (RAG)-based LFQA system, comprising an embedder, retriever,
reranker, and generator. A multi-faceted evaluation approach, including human
ranking, automatic metrics, and GPT-4 scoring, was employed to benchmark the
performance of different LFQA system configurations under heightened noisy
conditions. The results indicate that: (1) Among all compared generators,
Baichuan2-7B competes closely with GPT-3.5-turbo in accuracy score; (2) The
most effective system configuration on our dataset involved setting the
embedder, retriever, reranker, and generator as Ada2, Automated Merged
Retrieval, Bge-Reranker-Base, and Baichuan2-7B, respectively; (3) models are
less susceptible to noise after the length of contexts reaching a specific
threshold.

该研究介绍了 FinTextQA，这是一个用于金融领域长篇问答的创新数据集，并开发了一个基于 RAG 的 LFQA 系统，通过多角度评估方法得出了在噪声环境下不同配置的 LFQA 系统的性能表现和模型对噪声的容忍度。