Large language models (LLMs) have achieved impressive human-like performance
across various reasoning tasks. However, their mastery of underlying
inferential rules still falls short of human capabilities. To investigate this,
we propose a logic scaffolding inferential rule generation framework, to
construct an inferential rule base, ULogic, comprising both primitive and
compositional rules across five domains. Our analysis of GPT-series models over
a rule subset reveals significant gaps in LLMs' logic understanding compared to
human performance, especially in compositional and structural complex rules
with certain bias patterns. We further distill these rules into a smaller-scale
inference engine for flexible rule generation and enhancing downstream
reasoning. Through a multi-judger evaluation, our inference engine proves
effective in generating accurate, complex and abstract conclusions and
premises, and improve various commonsense reasoning tasks. Overall, our work
sheds light on LLMs' limitations in grasping inferential rule and suggests ways
to enhance their logical reasoning abilities~\footnote{Code and data are
available at https://github.com/SiyuanWangw/ULogic.}.

通过基于逻辑搭建的推理规则生成框架 ULgogic 对 GPT 系列模型进行分析，揭示了它们在逻辑理解方面与人类表现相比存在的显著差距，尤其是在具有某种偏见模式的复合和结构复杂规则方面；同时，我们采用这些规则构建了一个小规模推理引擎，用于灵活地生成规则并增强下游推理能力，经过多评估者评估，我们的推理引擎证明在生成准确、复杂和抽象的结论和前提方面非常有效，从而为增强大型语言模型的逻辑推理能力提供了启示。