Formal logic has long been applied to natural language reasoning, but this approach can sometimes lead to conclusions that, while logically entailed, are factually inconsistent with the premises or are not typically inferred by humans. This study introduces the concept of "rulebreakers", which refers to instances where logical entailment diverges from factually acceptable inference. We present RULEBREAKERS, a novel dataset for evaluating Large Language Models' (LLMs) ability to distinguish between rulebreakers and non-rulebreakers. Focusing on modus tollens and disjunctive syllogism, we assess six state-of-the-art LLMs using RULEBREAKERS, measuring their performance in terms of token-level exact accuracy and model confidence. Our findings reveal that while most models perform poorly to moderately in recognizing rulebreakers, they demonstrate a latent ability to distinguish rulebreakers when assessed by their confidence levels. Further analysis suggests that the failure to recognize rulebreakers is potentially associated with the models' world knowledge and their attention distribution patterns. This research highlights the limitation of LLMs' reasoning capabilities, and contributes to the ongoing discussion on reasoning in LLMs.

本研究针对大型语言模型在推理中存在的假设与现实之间的矛盾问题，提出了“规则破坏者”的概念，并构建了RULEBREAKERS数据集来评估模型的识别能力。研究发现，大多数模型在识别规则破坏者的能力上表现不佳，但通过评估其自信度，可以看出它们有潜力区分这些规则破坏者，从而揭示了大型语言模型推理能力的局限性。

规则破坏者挑战：揭示大型语言模型推理中盲点