Recent scholarship on reasoning in LLMs has supplied evidence of impressive
performance and flexible adaptation to machine generated or human feedback.
Nonmonotonic reasoning, crucial to human cognition for navigating the real
world, remains a challenging, yet understudied task. In this work, we study
nonmonotonic reasoning capabilities of seven state-of-the-art LLMs in one
abstract and one commonsense reasoning task featuring generics, such as 'Birds
fly', and exceptions, 'Penguins don't fly' (see Fig. 1). While LLMs exhibit
reasoning patterns in accordance with human nonmonotonic reasoning abilities,
they fail to maintain stable beliefs on truth conditions of generics at the
addition of supporting examples ('Owls fly') or unrelated information ('Lions
have manes'). Our findings highlight pitfalls in attributing human reasoning
behaviours to LLMs, as well as assessing general capabilities, while consistent
reasoning remains elusive.

最近的关于 LLMs 推理的学术研究提供了令人印象深刻的表现和对机器生成或人类反馈的灵活适应的证据。非单调推理对于人类认知来说至关重要，用于在现实世界中进行导航，但仍然是一个具有挑战性但研究不足的任务。我们研究了七种最先进的 LLMs 在一个抽象推理任务和一个常识推理任务中的非单调推理能力，这两个任务都涉及到如 “鸟会飞” 和 “企鹅不会飞” 等概括性陈述以及其例外情况。虽然 LLMs 表现出与人类非单调推理能力相符合的推理模式，但在支持性例子（“猫头鹰会飞”）或不相关信息（“狮子有鬃毛”）的添加时，它们无法保持对概括陈述的真实性条件的稳定信念。我们的研究结果突显了将人类推理行为归因于 LLMs 以及评估其总体能力的隐患，而一致的推理仍然难以实现。

LLM 是古典还是非单调推理者？从通性中得到的教训

Are LLMs classical or nonmonotonic reasoners? Lessons from generics

We present a new resource for the NLP community, namely a large (3.5M+
sentence) knowledge base of *generic statements*, e.g., "Trees remove carbon
dioxide from the atmosphere", collected from multiple corpora. This is the
first large resource to contain *naturally occurring* generic sentences, as
opposed to extracted or crowdsourced triples, and thus is rich in high-quality,
general, semantically complete statements. All GenericsKB sentences are
annotated with their topical term, surrounding context (sentences), and a
(learned) confidence. We also release GenericsKB-Best (1M+ sentences),
containing the best-quality generics in GenericsKB augmented with selected,
synthesized generics from WordNet and ConceptNet. In tests on two existing
datasets requiring multihop reasoning (OBQA and QASC), we find using GenericsKB
can result in higher scores and better explanations than using a much larger
corpus. This demonstrates that GenericsKB can be a useful resource for NLP
applications, as well as providing data for linguistic studies of generics and
their semantics. GenericsKB is available at
this https URL

我们提供了一个新的资源 —— 大型（3.5M + 句子）通用语句知识库，它包含从多个语料库中收集的自然生成的语句。这是第一个包含自然生成通用语句的大型资源，同时富含优质、通用、语义完整的语句。在两个需要多跳推理的现有数据集（OBQA 和 QASC）的测试中，使用 GenericsKB 可以得到比使用更大语料库更高的得分和更好的解释。这表明 GenericsKB 不仅可以成为 NLP 应用的有用资源，还可以为通用语句及其语义的语言研究提供数据。