The reasoning abilities of Large Language Models (LLMs) are becoming a
central focus of study in NLP. In this paper, we consider the case of
syllogistic reasoning, an area of deductive reasoning studied extensively in
logic and cognitive psychology. Previous research has shown that pre-trained
LLMs exhibit reasoning biases, such as $\textit{content effects}$, avoid
answering that $\textit{no conclusion follows}$, display human-like
difficulties, and struggle with multi-step reasoning. We contribute to this
research line by systematically investigating the effects of chain-of-thought
reasoning, in-context learning (ICL), and supervised fine-tuning (SFT) on
syllogistic reasoning, considering syllogisms with conclusions that support or
violate world knowledge, as well as ones with multiple premises. Crucially, we
go beyond the standard focus on accuracy, with an in-depth analysis of the
conclusions generated by the models. Our results suggest that the behavior of
pre-trained LLMs can be explained by heuristics studied in cognitive science
and that both ICL and SFT improve model performance on valid inferences,
although only the latter mitigates most reasoning biases without harming model
consistency.

我们系统地研究了大型语言模型在选择式推理中的思维方式、背景学习和有监督微调的影响，并考虑了支持或违反世界知识的结论以及具有多个前提的典型推理。我们的结果表明，大型语言模型的行为可以通过认知科学中研究的启发式方式来解释，而背景学习和有监督微调都能提高模型在有效推理上的性能，尽管只有有监督微调能够减少大部分的推理偏差而不损害模型的一致性。

大型语言模型作为软性推理器的系统分析：以三段论推理为例

A Systematic Analysis of Large Language Models as Soft Reasoners: The  Case of Syllogistic Inferences

The success of Large Language Models (LLMs), e.g., ChatGPT, is witnessed by
their planetary popularity, their capability of human-like question-answering,
and also by their steadily improved reasoning performance. However, it remains
unclear whether LLMs reason. It is an open problem how traditional neural
networks can be qualitatively extended to go beyond the statistic paradigm and
achieve high-level cognition. Here, we present a minimalist qualitative
extension by generalising computational building blocks from vectors to
spheres. We propose Sphere Neural Networks (SphNNs) for human-like reasoning
through model construction and inspection, and develop SphNN for syllogistic
reasoning, a microcosm of human rationality. Instead of training data, SphNN
uses a neuro-symbolic transition map of neighbourhood spatial relations to
guide transformations from the current sphere configuration towards the target.
SphNN is the first neural model that can determine the validity of long-chained
syllogistic reasoning in one epoch by constructing sphere configurations as
Euler diagrams, with the worst computational complexity of O(N^2). SphNN can
evolve into various types of reasoning, such as spatio-temporal reasoning,
logical reasoning with negation and disjunction, event reasoning,
neuro-symbolic reasoning, and humour understanding (the highest level of
cognition). All these suggest a new kind of Herbert A. Simon's scissors with
two neural blades. SphNNs will tremendously enhance interdisciplinary
collaborations to develop the two neural blades and realise deterministic
neural reasoning and human-bounded rationality and elevate LLMs to reliable
psychological AI. This work suggests that the non-zero radii of spheres are the
missing components that prevent traditional deep-learning systems from reaching
the realm of rational reasoning and cause LLMs to be trapped in the swamp of
hallucination.

通过在计算构件中将向量泛化为球体，我们提出了球形神经网络（SphNNs）进行人类级推理，包括主义推理、逻辑推理、事件推理、神经符号推理和幽默理解。这项工作认为，球体的非零半径是阻止传统深度学习系统进入理性推理领域，并使得大型语言模型在幻觉的泥沼中陷入困境的缺失组件。

球面神经网络用于合理推理

Sphere Neural-Networks for Rational Reasoning

This paper investigates whether current large language models exhibit biases
in logical reasoning, similar to humans. Specifically, we focus on syllogistic
reasoning, a well-studied form of inference in the cognitive science of human
deduction. To facilitate our analysis, we introduce a dataset called NeuBAROCO,
originally designed for psychological experiments that assess human logical
abilities in syllogistic reasoning. The dataset consists of syllogistic
inferences in both English and Japanese. We examine three types of biases
observed in human syllogistic reasoning: belief biases, conversion errors, and
atmosphere effects. Our findings demonstrate that current large language models
struggle more with problems involving these three types of biases.

本文研究当前的大型语言模型是否存在逻辑推理的偏见，探讨三种主要偏见类型，并介绍一种心理实验数据集 NeuBAROCO。通过分析数据集，发现当前的大型语言模型在涉及到这三种类型偏见的问题上表现欠佳。