The reasoning abilities of Large Language Models (LLMs) are becoming a
central focus of study in NLP. In this paper, we consider the case of
syllogistic reasoning, an area of deductive reasoning studied extensively in
logic and cognitive psychology. Previous research has shown that pre-trained
LLMs exhibit reasoning biases, such as $\textit{content effects}$, avoid
answering that $\textit{no conclusion follows}$, display human-like
difficulties, and struggle with multi-step reasoning. We contribute to this
research line by systematically investigating the effects of chain-of-thought
reasoning, in-context learning (ICL), and supervised fine-tuning (SFT) on
syllogistic reasoning, considering syllogisms with conclusions that support or
violate world knowledge, as well as ones with multiple premises. Crucially, we
go beyond the standard focus on accuracy, with an in-depth analysis of the
conclusions generated by the models. Our results suggest that the behavior of
pre-trained LLMs can be explained by heuristics studied in cognitive science
and that both ICL and SFT improve model performance on valid inferences,
although only the latter mitigates most reasoning biases without harming model
consistency.

我们系统地研究了大型语言模型在选择式推理中的思维方式、背景学习和有监督微调的影响，并考虑了支持或违反世界知识的结论以及具有多个前提的典型推理。我们的结果表明，大型语言模型的行为可以通过认知科学中研究的启发式方式来解释，而背景学习和有监督微调都能提高模型在有效推理上的性能，尽管只有有监督微调能够减少大部分的推理偏差而不损害模型的一致性。

大型语言模型作为软性推理器的系统分析：以三段论推理为例

A Systematic Analysis of Large Language Models as Soft Reasoners: The  Case of Syllogistic Inferences

The development of highly fluent large language models (LLMs) has prompted
increased interest in assessing their reasoning and problem-solving
capabilities. We investigate whether several LLMs can solve a classic type of
deductive reasoning problem from the cognitive science literature. The tested
LLMs have limited abilities to solve these problems in their conventional form.
We performed follow up experiments to investigate if changes to the
presentation format and content improve model performance. We do find
performance differences between conditions; however, they do not improve
overall performance. Moreover, we find that performance interacts with
presentation format and content in unexpected ways that differ from human
performance. Overall, our results suggest that LLMs have unique reasoning
biases that are only partially predicted from human reasoning performance.

大型语言模型（LLMs）的发展促使人们对其推理和问题解决能力产生了更大的兴趣。本研究调查了几种 LLMs 是否能够解决认知科学文献中一种经典类型的演绎推理问题。研究发现，这些被测试的 LLMs 在传统形式上解决这些问题的能力有限。我们进行了后续实验，探究了更改展示格式和内容是否能改善模型性能。尽管我们发现了条件之间的绩效差异，但总体性能并未提高。此外，我们还发现性能与展示格式和内容以出人意料的方式相互作用，与人类表现有所不同。总的来说，我们的结果表明 LLMs 具有独特的推理偏见，其只能部分预测人类的推理表现。