Recently, large language models (LLMs), including notable models such as
GPT-4 and burgeoning community models, have showcased significant general
language understanding abilities. However, there has been a scarcity of
attempts to assess the logical reasoning capacities of these LLMs, an essential
facet of natural language understanding. To encourage further investigation in
this area, we introduce GLoRE, a meticulously assembled General Logical
Reasoning Evaluation benchmark comprised of 12 datasets that span three
different types of tasks. Our experimental results show that compared to the
performance of human and supervised fine-tuning, the logical reasoning
capabilities of open LLM models necessitate additional improvement; ChatGPT and
GPT-4 show a strong capability of logical reasoning, with GPT-4 surpassing
ChatGPT by a large margin. We propose a self-consistency probing method to
enhance the accuracy of ChatGPT and a fine-tuned method to boost the
performance of an open LLM. We release the datasets and evaluation programs to
facilitate future research.

该篇研究论文主要介绍了大型语言模型的逻辑推理能力评估，提出了一个名为 GLoRE 的评估基准，包含 12 个数据集，通过实验证明了 ChatGPT 和 GPT-4 在逻辑推理方面的强大能力，并提出了改进方法，发布了数据集和评估程序以促进未来的研究。