Recent LLMs have demonstrated remarkable performance in solving exam-like
math word problems. However, the degree to which these numerical reasoning
skills are effective in real-world scenarios, particularly in expert domains,
is still largely unexplored. This paper introduces DocMath-Eval, a
comprehensive benchmark specifically designed to evaluate the numerical
reasoning and problem-solving capabilities of LLMs in the context of
understanding and analyzing financial documents containing both text and
tables. We evaluate a wide spectrum of 19 LLMs, including those specialized in
coding and finance. We also incorporate different prompting strategies (i.e.,
Chain-of-Thoughts and Program-of-Thoughts) to comprehensively assess the
capabilities and limitations of existing LLMs in DocMath-Eval. We found that,
although the current best-performing system (i.e., GPT-4), can perform well on
simple problems such as calculating the rate of increase in a financial metric
within a short document context, it significantly lags behind human experts in
more complex problems grounded in longer contexts. We believe DocMath-Eval can
be used as a valuable benchmark to evaluate LLMs' capabilities to solve
challenging numerical reasoning problems in expert domains. We will release the
benchmark and code at this https URL

该论文介绍了 DocMath-Eval，这是一个特意设计用于评估具有理解和分析既包含文本又包含表格的金融文件的 LLMs 的数值推理和问题解决能力的综合基准测试。我们评估了 19 个 LLMs 的广泛领域的表现，并采用不同的提示策略来全面评估现有 LLMs 在 DocMath-Eval 中的能力和局限性。我们发现，尽管目前表现最好的系统（即 GPT-4）在短文本环境下可以很好地处理简单问题，如计算金融指标的增长率，但在较长上下文中的更复杂问题上明显落后于人类专家。我们相信 DocMath-Eval 可以作为一个有价值的基准测试，用于评估 LLMs 在专家领域中解决具有挑战性的数值推理问题的能力。我们将在此 https URL 上发布该基准测试和代码。