As Large Language Models (LLMs) continue to exhibit remarkable performance in
natural language understanding tasks, there is a crucial need to measure their
ability for human-like multi-step logical reasoning. Existing logical reasoning
evaluation benchmarks often focus primarily on simplistic single-step or
multi-step reasoning with a limited set of inference rules. Furthermore, the
lack of datasets for evaluating non-monotonic reasoning represents a crucial
gap since it aligns more closely with human-like reasoning. To address these
limitations, we propose Multi-LogiEval, a comprehensive evaluation dataset
encompassing multi-step logical reasoning with various inference rules and
depths. Multi-LogiEval covers three logic types--propositional, first-order,
and non-monotonic--consisting of more than 30 inference rules and more than 60
of their combinations with various depths. Leveraging this dataset, we conduct
evaluations on a range of LLMs including GPT-4, ChatGPT, Gemini-Pro, Yi, Orca,
and Mistral, employing a zero-shot chain-of-thought. Experimental results show
that there is a significant drop in the performance of LLMs as the reasoning
steps/depth increases (average accuracy of ~68% at depth-1 to ~43% at depth-5).
We further conduct a thorough investigation of reasoning chains generated by
LLMs which reveals several important findings. We believe that Multi-LogiEval
facilitates future research for evaluating and enhancing the logical reasoning
ability of LLMs. Data is available at
this https URL

为了评估大型语言模型在人类式的多步逻辑推理方面的能力，我们提出了一种综合评估数据集 Multi-LogiEval，该数据集囊括了多步逻辑推理、各种推理规则和深度，并覆盖了命题逻辑、一阶逻辑和非单调逻辑等三种逻辑类型。实验结果表明，随着推理步骤 / 深度的增加，语言模型的性能显著下降（在深度为 1 时平均准确率约为 68%，深度为 5 时约为 43%）。我们相信 Multi-LogiEval 将有助于以后评估和提升大型语言模型的逻辑推理能力。

多阶段逻辑推理能力评估的目标：Multi-LogiEval

Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability  of Large Language Models

Language models have been shown to perform remarkably well on a wide range of
natural language processing tasks. In this paper, we propose a novel system
that uses language models to perform multi-step logical reasoning. Our system
incorporates explicit planning into its inference procedure, thus able to make
more informed reasoning decisions at each step by looking ahead into their
future effects. In our experiments, our full system significantly outperforms
other competing systems. On a multiple-choice question answering task, our
system performs competitively compared to GPT-3-davinci despite having only
around 1.5B parameters. We conduct several ablation studies to demonstrate that
explicit planning plays a crucial role in the system's performance.

在本文中，我们提出了一个使用语言模型进行多步逻辑推理的新型系统，该系统将显式计划纳入其推理过程中，从而通过向前看到它们的未来效果，能够在每一步上做出更明智的推理决策。在我们的实验中，我们的完整系统明显优于其他竞争系统，在多项选择题回答任务中，我们的系统表现与 GPT-3-davinci 相当，尽管只有约 15B 的参数。我们进行了几项削减研究，以证明显式计划在系统性能中起着重要作用。