This study intends to systematically disentangle pure logic reasoning and
text understanding by investigating the contrast across abstract and
contextualized logical problems from a comprehensive set of domains. We explore
whether LLMs demonstrate genuine reasoning capabilities across various domains
when the underlying logical structure remains constant. We focus on two main
questions (1) Can abstract logical problems alone accurately benchmark an LLM's
reasoning ability in real-world scenarios, disentangled from contextual support
in practical settings? (2) Does fine-tuning LLMs on abstract logic problem
generalize to contextualized logic problems and vice versa? To investigate
these questions, we focus on standard propositional logic, specifically
propositional deductive and abductive logic reasoning. In particular, we
construct instantiated datasets for deductive and abductive reasoning with 4
levels of difficulty, encompassing 12 distinct categories or domains based on
the categorization of Wikipedia. Our experiments aim to provide insights into
disentangling context in logical reasoning and the true reasoning capabilities
of LLMs and their generalization potential. The code and dataset are available
at: this https URL

通过调查来自一系列领域中的抽象和上下文逻辑问题的对比，本研究旨在系统地分离纯逻辑推理和文本理解。我们探讨当基础逻辑结构保持不变时，LLMs 在各个领域中是否展示出真正的推理能力。