Algorithmic reasoning is a fundamental cognitive ability that plays a pivotal
role in problem-solving and decision-making processes. Reinforcement Learning
(RL) has demonstrated remarkable proficiency in tasks such as motor control,
handling perceptual input, and managing stochastic environments. These
advancements have been enabled in part by the availability of benchmarks. In
this work we introduce PUZZLES, a benchmark based on Simon Tatham's Portable
Puzzle Collection, aimed at fostering progress in algorithmic and logical
reasoning in RL. PUZZLES contains 40 diverse logic puzzles of adjustable sizes
and varying levels of complexity; many puzzles also feature a diverse set of
additional configuration parameters. The 40 puzzles provide detailed
information on the strengths and generalization capabilities of RL agents.
Furthermore, we evaluate various RL algorithms on PUZZLES, providing baseline
comparisons and demonstrating the potential for future research. All the
software, including the environment, is available at
this https URL

算法推理在问题解决和决策过程中起着重要作用，强化学习在运动控制、处理感知输入和管理随机环境等任务中表现出卓越的技能。本研究引入了 PUZZLES，这是一个基于 Simon Tatham 的便携式拼图集的基准，旨在促进算法和逻辑推理在强化学习中的进展。PUZZLES 包含 40 个不同大小和复杂程度的多样逻辑拼图，许多拼图还具有一组多样的配置参数。这 40 个拼图提供了关于强化学习代理的优势和泛化能力的详细信息。此外，我们在 PUZZLES 上评估了各种强化学习算法，提供了基准比较，并展示了未来研究的潜力。所有软件和环境都可在此 https URL 网址上获得。

PUZZLES：神经算法推理的基准

PUZZLES: A Benchmark for Neural Algorithmic Reasoning

I shall quantify the logical faults generated by ChatGPT when applied to
reasoning tasks. For experiments, I use the 144 puzzles from the library
https://users.utcluj.ro/~agroza/puzzles/maloga~\cite{groza:fol}. The
library contains puzzles of various types, including arithmetic puzzles,
logical equations, Sudoku-like puzzles, zebra-like puzzles, truth-telling
puzzles, grid puzzles, strange numbers, or self-reference puzzles. The correct
solutions for these puzzles were checked using the theorem prover
Prover9~\cite{mccune2005release} and the finite models finder
Mace4~\cite{mccune2003mace4} based on human-modelling in Equational First Order
Logic. A first output of this study is the benchmark of 100 logical puzzles.
For this dataset ChatGPT provided both correct answer and justification for 7\%
only. %, while BARD for 5\%. Since the dataset seems challenging, the
researchers are invited to test the dataset on more advanced or tuned models
than ChatGPT3.5 with more crafted prompts. A second output is the
classification of reasoning faults conveyed by ChatGPT. This classification
forms a basis for a taxonomy of reasoning faults generated by large language
models. I have identified 67 such logical faults, among which: inconsistencies,
implication does not hold, unsupported claim, lack of commonsense, wrong
justification. The 100 solutions generated by ChatGPT contain 698 logical
faults. That is on average, 7 fallacies for each reasoning task. A third ouput
is the annotated answers of the ChatGPT with the corresponding logical faults.
Each wrong statement within the ChatGPT answer was manually annotated, aiming
to quantify the amount of faulty text generated by the language model. On
average, 26.03\% from the generated text was a logical fault.

ChatGPT 应用于推理任务时产生的逻辑错误的量化研究，包括逻辑错误的分类和基于大型语言模型产生的推理错误的标注答案。