We introduce a large language model (LLM) based approach to answer complex
questions requiring multi-hop numerical reasoning over financial reports. While
LLMs have exhibited remarkable performance on various natural language and
reasoning tasks, complex reasoning problems often rely on few-shot prompts that
require carefully crafted examples. In contrast, our approach uses novel
zero-shot prompts that guide the LLM to encode the required reasoning into a
Python program or a domain specific language. The generated program is then
executed by a program interpreter, thus mitigating the limitations of LLM in
performing accurate arithmetic calculations.
We evaluate the proposed approach on three financial datasets using some of
the recently developed generative pretrained transformer (GPT) models and
perform comparisons with various zero-shot baselines. The experimental results
demonstrate that our approach significantly improves the accuracy for all the
LLMs over their respective baselines. We provide a detailed analysis of the
results, generating insights to support our findings. The success of our
approach demonstrates the enormous potential to extract complex domain specific
numerical reasoning by designing zero-shot prompts to effectively exploit the
knowledge embedded in LLMs.

我们引入了一种基于大型语言模型的方法来回答需要多跳数字推理的复杂问题。我们的方法使用新颖的零样本提示，将所需的推理编码到 Python 程序或领域特定语言中，并通过程序解释器执行生成的程序，从而减轻了大型语言模型在执行准确的算术计算时的局限性。我们在三个财务数据集上评估了所提出的方法，使用一些最近开发的生成式预训练变压器模型，并与各种零样本基准进行比较。实验结果表明，我们的方法显著提高了大型语言模型在所有基线上的准确性。我们对结果进行了详细分析，生成支持我们发现的洞察。我们的方法的成功证明，通过设计有效利用嵌入在大型语言模型中的知识的零样本提示，可以提取复杂的领域特定数值推理的巨大潜力。