Recent works have explored the use of LLMs for reasoning tasks focussing on
relatively simple problems, such as logical question answering. In our work, we
wish to tackle more complicated problems, significantly expanding the
capabilities of these models. Particularly, we explore whether LLMs can solve
challenging first-order combinatorial reasoning problems, an example being the
popular puzzle Sudoku. These problems have an underlying first-order structure
described by a general description in natural language and can be instantiated
to instances of varying sizes. Moreover these problems are computationally
intensive requiring several reasoning steps to reach the solution. We present
PuzzleBench a dataset of 31 such challenging puzzles. We observe that LLMs even
when aided by symbolic solvers perform rather poorly on our benchmark. In
response we propose a new approach, Puzzle-LM which combines LLMs with both
symbolic solvers and program interpreters enabling them to reason about such
challenging problems. We also show how feedback from smaller solved instances
can help improve this reasoning ability.

我们工作的目标是探索语言模型是否可以解决复杂的一阶组合推理问题，例如数独，尽管现有的 LLMs 在这些问题上表现较差。因此，我们提出了一种新的方法，Puzzle-LM，将 LLMs 与符号求解器和程序解释器相结合，以改善解决这些问题的能力。我们还展示了通过解决较小的实例获得的反馈如何提高推理能力。