Logical reasoning serves as a cornerstone for human cognition. Recently, the emergence of Large Language Models (LLMs) has demonstrated promising progress in solving logical reasoning tasks effectively. To improve this capability, recent studies have delved into integrating LLMs with various symbolic solvers using diverse techniques and methodologies. While some combinations excel on specific datasets, others fall short. However, it remains unclear whether the variance in performance stems from the methodologies employed or the specific symbolic solvers utilized. Therefore, there is a lack of consistent comparison between symbolic solvers and how they influence LLM's logical reasoning ability. We perform experiments on LLMs integrated with 3 symbolic solvers: Z3, Pyke, and Prover9, and compare their performance on 3 logical reasoning datasets: ProofWriter, PrOntoQA, and FOLIO. Our findings indicate that when combined with LLMs Pyke's performance is significantly inferior to that of Prover9 and Z3. Z3's overall accuracy performance slightly surpasses Prover9, but Prover9 could execute more questions.

通过将大型语言模型 (LLMs) 与各种符号求解器相结合，我们对 Z3、Pyke 和 Prover9 三个符号求解器的性能进行实验证明，其中与 LLMs 相结合时，Pyke 的性能明显低于 Prover9 和 Z3，Z3 的总体准确性略高于 Prover9，但 Prover9 能够处理更多问题。

深度研究逻辑推理与LLM：工具选择的重要性