We introduce LLM-ARC, a neuro-symbolic framework designed to enhance the
logical reasoning capabilities of Large Language Models (LLMs), by combining
them with an Automated Reasoning Critic (ARC). LLM-ARC employs an Actor-Critic
method where the LLM Actor generates declarative logic programs along with
tests for semantic correctness, while the Automated Reasoning Critic evaluates
the code, runs the tests and provides feedback on test failures for iterative
refinement. Implemented using Answer Set Programming (ASP), LLM-ARC achieves a
new state-of-the-art accuracy of 88.32% on the FOLIO benchmark which tests
complex logical reasoning capabilities. Our experiments demonstrate significant
improvements over LLM-only baselines, highlighting the importance of logic test
generation and iterative self-refinement. We achieve our best result using a
fully automated self-supervised training loop where the Actor is trained on
end-to-end dialog traces with Critic feedback. We discuss potential
enhancements and provide a detailed error analysis, showcasing the robustness
and efficacy of LLM-ARC for complex natural language reasoning tasks.

LLM-ARC 是一个神经符号框架，通过将大语言模型（LLMs）与自动推理评论家（ARC）相结合，提升其逻辑推理能力。它采用了演员 - 评论家方法，其中 LLM 演员生成声明性逻辑程序和语义正确性测试，而自动推理评论家评估代码，运行测试，并提供关于测试失败的反馈以进行迭代改进。在 FOLIO 基准测试中，利用 ASP 实现的 LLM-ARC 实现了 88.32% 的新的最高准确性，测试了复杂的逻辑推理能力。我们的实验证明了 LLM-ARC 相对于仅使用 LLM 的基准线的显著改进，突出了逻辑测试生成和迭代自我改进的重要性。我们通过完全自动化的自我监督训练循环达到了最佳结果，其中演员在端到端对话追踪上接受评论家的反馈训练。我们讨论了潜在的改进，并提供了详细的错误分析，展示了 LLM-ARC 在复杂自然语言推理任务中的鲁棒性和有效性。