We introduce a large language model (LLM) based approach to answer complex
questions requiring multi-hop numerical reasoning over financial reports. While
LLMs have exhibited remarkable performance on various natural language and
reasoning tasks, complex reasoning problems often rely on few-shot prompts that
require carefully crafted examples. In contrast, our approach uses novel
zero-shot prompts that guide the LLM to encode the required reasoning into a
Python program or a domain specific language. The generated program is then
executed by a program interpreter, thus mitigating the limitations of LLM in
performing accurate arithmetic calculations.
We evaluate the proposed approach on three financial datasets using some of
the recently developed generative pretrained transformer (GPT) models and
perform comparisons with various zero-shot baselines. The experimental results
demonstrate that our approach significantly improves the accuracy for all the
LLMs over their respective baselines. We provide a detailed analysis of the
results, generating insights to support our findings. The success of our
approach demonstrates the enormous potential to extract complex domain specific
numerical reasoning by designing zero-shot prompts to effectively exploit the
knowledge embedded in LLMs.

我们引入了一种基于大型语言模型的方法来回答需要多跳数字推理的复杂问题。我们的方法使用新颖的零样本提示，将所需的推理编码到 Python 程序或领域特定语言中，并通过程序解释器执行生成的程序，从而减轻了大型语言模型在执行准确的算术计算时的局限性。我们在三个财务数据集上评估了所提出的方法，使用一些最近开发的生成式预训练变压器模型，并与各种零样本基准进行比较。实验结果表明，我们的方法显著提高了大型语言模型在所有基线上的准确性。我们对结果进行了详细分析，生成支持我们发现的洞察。我们的方法的成功证明，通过设计有效利用嵌入在大型语言模型中的知识的零样本提示，可以提取复杂的领域特定数值推理的巨大潜力。

基于大型语言模型的金融文件零样本问答

Zero-Shot Question Answering over Financial Documents using Large  Language Models

Large language models have demonstrated remarkable performance across various
natural language processing tasks; however, their efficacy in more challenging
and domain-specific tasks remains less explored. This paper introduces the
GAOKAO-Benchmark (GAOKAO-Bench), an intuitive benchmark that employs questions
from the Chinese Gaokao examination as test samples for evaluating large
language models.In order to align the evaluation results with humans as much as
possible, we designed a method based on zero-shot prompts to analyze the
accuracy and scoring rate of the model by dividing the questions into
subjective and objective types. We evaluated the ChatGPT model on
GAOKAO-Benchmark performance.Our findings reveal that the ChatGPT model excels
in tackling objective questions, while also shedding light on its shortcomings
and areas for improvement. To further scrutinize the model's responses, we
incorporate human evaluations.In conclusion, this research contributes a robust
evaluation benchmark for future large-scale language models and offers valuable
insights into the limitations of such models.

这篇论文介绍了 GAOKAO-Benchmark，它是一个直观的基准测试，利用中国高考考试的问题作为测试样本，对大型语言模型进行评估的一种方法。我们采用基于零 - shot prompts 的方法来分析模型的准确率和评分率，发现 ChatGPT 模型在解决客观问题方面表现优异，同时也揭示了它的不足之处和改进方向，为未来大规模语言模型的评估提供了坚实的基础和有价值的见解。

评估大型语言模型在高考基准测试上的表现

Evaluating the Performance of Large Language Models on GAOKAO Benchmark

Emergent chain-of-thought (CoT) reasoning capabilities promise to improve
performance and explainability of large language models (LLMs). However,
uncertainties remain about how prompting strategies formulated for previous
model generations generalize to new model generations and different datasets.
In this small-scale study we compare the performance of a range of zero-shot
prompts for inducing CoT reasoning across six recently released LLMs
(davinci-002, davinci-003, GPT-3.5-turbo, GPT-4, Flan-T5-xxl and Cohere
command-xlarge) on a mixture of six question-answering datasets, including
datasets from scientific and medical domains. We find that a CoT prompt that
was previously discovered through automated prompt discovery shows robust
performance across experimental conditions and produces best results when
applied to the state-of-the-art model GPT-4.

通过使用自动探索的 CoT prompt 对多个 LLMs 进行无监督语言生成，我们发现将该 prompt 应用于最新的 GPT-4 模型是最优的。