Large Language Models (LLMs) have shown remarkable proficiency in language
understanding and have been successfully applied to a variety of real-world
tasks through task-specific fine-tuning or prompt engineering. Despite these
advancements, it remains an open question whether LLMs are fundamentally
capable of reasoning and planning, or if they primarily rely on recalling and
synthesizing information from their training data. In our research, we
introduce a novel task -- Minesweeper -- specifically designed in a format
unfamiliar to LLMs and absent from their training datasets. This task
challenges LLMs to identify the locations of mines based on numerical clues
provided by adjacent opened cells. Successfully completing this task requires
an understanding of each cell's state, discerning spatial relationships between
the clues and mines, and strategizing actions based on logical deductions drawn
from the arrangement of the cells. Our experiments, including trials with the
advanced GPT-4 model, indicate that while LLMs possess the foundational
abilities required for this task, they struggle to integrate these into a
coherent, multi-step logical reasoning process needed to solve Minesweeper.
These findings highlight the need for further research to understand and nature
of reasoning capabilities in LLMs under similar circumstances, and to explore
pathways towards more sophisticated AI reasoning and planning models.

我们的研究引入了一种新的任务 -- 扫雷，旨在测试 LLMs 在陌生格式的任务中的推理和规划能力；我们的实验证明，尽管 LLMs 具备完成该任务所需的基本能力，但它们在将这些能力整合成解决扫雷问题所需的连贯的多步骤逻辑推理过程方面存在困难。这些发现强调了进一步研究 LLMs 推理能力及探索更复杂的 AI 推理和规划模型的必要性。

大型语言模型中的逻辑谜题解决评估：基于扫雷案例研究的洞察

Assessing Logical Puzzle Solving in Large Language Models: Insights from  a Minesweeper Case Study

Minesweeper is an interesting single player game based on logic, memory and
guessing. Solving Minesweeper has been shown to be an NP-hard task.
Deterministic solvers are the best known approach for solving Minesweeper. This
project proposes a neural network based learner for solving Minesweeper. To
choose the best learner, different architectures and configurations of neural
networks were trained on hundreds of thousands of games. Surprisingly, the
proposed neural network based learner has shown to be a very good approximation
function for solving Minesweeper. The neural network learner competes well with
the CSP solvers, especially in Beginner and Intermediate modes of the game. It
was also observed that despite having high success rates, the best neural
learner was considerably slower than the best deterministic solver. This report
also discusses the overheads and limitations faced while creating highly
successful neural networks for Minesweeper.

该研究提出了基于神经网络的 Minesweeper 学习器，该学习器具有很好的求解效果，并表现出与 CSP 求解器相竞争的能力。然而，神经学习器的运行速度远远不及确定性求解器，并且在创建高度成功的神经网络时面临诸多开销和限制。