Large language models (LLMs) have achieved impressive human-like performance
across various reasoning tasks. However, their mastery of underlying
inferential rules still falls short of human capabilities. To investigate this,
we propose a logic scaffolding inferential rule generation framework, to
construct an inferential rule base, ULogic, comprising both primitive and
compositional rules across five domains. Our analysis of GPT-series models over
a rule subset reveals significant gaps in LLMs' logic understanding compared to
human performance, especially in compositional and structural complex rules
with certain bias patterns. We further distill these rules into a smaller-scale
inference engine for flexible rule generation and enhancing downstream
reasoning. Through a multi-judger evaluation, our inference engine proves
effective in generating accurate, complex and abstract conclusions and
premises, and improve various commonsense reasoning tasks. Overall, our work
sheds light on LLMs' limitations in grasping inferential rule and suggests ways
to enhance their logical reasoning abilities~\footnote{Code and data are
available at https://github.com/SiyuanWangw/ULogic.}.

通过基于逻辑搭建的推理规则生成框架 ULgogic 对 GPT 系列模型进行分析，揭示了它们在逻辑理解方面与人类表现相比存在的显著差距，尤其是在具有某种偏见模式的复合和结构复杂规则方面；同时，我们采用这些规则构建了一个小规模推理引擎，用于灵活地生成规则并增强下游推理能力，经过多评估者评估，我们的推理引擎证明在生成准确、复杂和抽象的结论和前提方面非常有效，从而为增强大型语言模型的逻辑推理能力提供了启示。

LLMs 是否能够运用推理规则？用于压力测试和改善 LLMs 的逻辑支撑

Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and  Improving LLMs

In this work, we study whether multilingual language models (MultiLMs) can
transfer logical reasoning abilities to other languages when they are
fine-tuned for reasoning in a different language. We evaluate the cross-lingual
reasoning abilities of MultiLMs in two schemes: (1) where the language of the
context and the question remain the same in the new languages that are tested
(i.e., the reasoning is still monolingual, but the model must transfer the
learned reasoning ability across languages), and (2) where the language of the
context and the question is different (which we term code-switched reasoning).
On two logical reasoning datasets, RuleTaker and LeapOfThought, we demonstrate
that although MultiLMs can transfer reasoning ability across languages in a
monolingual setting, they struggle to transfer reasoning abilities in a
code-switched setting. Following this observation, we propose a novel attention
mechanism that uses a dedicated set of parameters to encourage cross-lingual
attention in code-switched sequences, which improves the reasoning performance
by up to 14% and 4% on the RuleTaker and LeapOfThought datasets, respectively.

本研究探讨了多语言语言模型（MultiLMs）在针对不同语言推理时，是否能够将逻辑推理能力转移到其他语言。通过在两种方案中评估 MultiLMs 的跨语言推理能力，我们发现在单语言设置下，MultiLMs 可以在语言之间传递推理能力，但在混合代码推理的情况下，它们很难传递推理能力。基于此观察，我们提出了一种新颖的注意机制，利用专门的参数集在混合代码序列中鼓励跨语言注意力，从而在 RuleTaker 和 LeapOfThought 数据集上分别将推理性能提高了 14% 和 4%。

打破语言障碍：通过结构化自注意力提升跨语言推理

Breaking the Language Barrier: Improving Cross-Lingual Reasoning with  Structured Self-Attention

Large language models (LLMs) take advantage of step-by-step reasoning
instructions, e.g., chain-of-thought (CoT) prompting. Building on this, their
ability to perform CoT-style reasoning robustly is of interest from a probing
perspective. In this study, we inspect the step-by-step reasoning ability of
LLMs with a focus on negation, which is a core linguistic phenomenon that is
difficult to process. In particular, we introduce several controlled settings
(e.g., reasoning in case of fictional entities) to evaluate the logical
reasoning abilities of the models. We observed that dozens of modern LLMs were
not robust against lexical negation (e.g., plausible ->implausible) when
performing CoT-style reasoning, and the results highlight unique limitations in
each LLM family.

大型语言模型在逐步推理指令（例如思维链提示）方面具有优势。在此基础上，它们在执行连贯思维推理方面的鲁棒性是一个值得探究的问题。本研究检验大型语言模型在逐步推理能力方面，重点关注难以处理的核心语言现象 —— 否定。通过引入几个受控设置（例如处理虚构实体时的推理）来评估模型的逻辑推理能力。我们观察到当进行逐步推理时，许多现代大型语言模型在处理否定词（例如 plausible -> implausible）时不具有鲁棒性，研究结果凸显了每个大型语言模型家族中独特的局限性。