FinanceBench is a first-of-its-kind test suite for evaluating the performance
of LLMs on open book financial question answering (QA). It comprises 10,231
questions about publicly traded companies, with corresponding answers and
evidence strings. The questions in FinanceBench are ecologically valid and
cover a diverse set of scenarios. They are intended to be clear-cut and
straightforward to answer to serve as a minimum performance standard. We test
16 state of the art model configurations (including GPT-4-Turbo, Llama2 and
Claude2, with vector stores and long context prompts) on a sample of 150 cases
from FinanceBench, and manually review their answers (n=2,400). The cases are
available open-source. We show that existing LLMs have clear limitations for
financial QA. Notably, GPT-4-Turbo used with a retrieval system incorrectly
answered or refused to answer 81% of questions. While augmentation techniques
such as using longer context window to feed in relevant evidence improve
performance, they are unrealistic for enterprise settings due to increased
latency and cannot support larger financial documents. We find that all models
examined exhibit weaknesses, such as hallucinations, that limit their
suitability for use by enterprises.

FinanceBench 是一个对 LLMs 在开放式财务问答（QA）中性能评估的首个测试套件，通过对公开交易公司的 10,231 个问题进行测试，发现现有的 LLMs 在财务 QA 方面存在明显的局限性。