Automated reasoning with unstructured natural text is a key requirement for
many potential applications of NLP and for developing robust AI systems.
Recently, Language Models (LMs) have demonstrated complex reasoning capacities
even without any finetuning. However, existing evaluation for automated
reasoning assumes access to a consistent and coherent set of information over
which models reason. When reasoning in the real-world, the available
information is frequently inconsistent or contradictory, and therefore models
need to be equipped with a strategy to resolve such conflicts when they arise.
One widely-applicable way of resolving conflicts is to impose preferences over
information sources (e.g., based on source credibility or information recency)
and adopt the source with higher preference. In this paper, we formulate the
problem of reasoning with contradictory information guided by preferences over
sources as the classical problem of defeasible reasoning, and develop a dataset
called BoardgameQA for measuring the reasoning capacity of LMs in this setting.
BoardgameQA also incorporates reasoning with implicit background knowledge, to
better reflect reasoning problems in downstream applications. We benchmark
various LMs on BoardgameQA and the results reveal a significant gap in the
reasoning capacity of state-of-the-art LMs on this problem, showing that
reasoning with conflicting information does not surface out-of-the-box in LMs.
While performance can be improved with finetuning, it nevertheless remains
poor.

该论文探讨了利用自然语言处理技术进行非结构化文本的自动推理。研究表明，语言模型具有复杂的推理能力。然而，当推理真实世界中不一致或矛盾的信息时，需要采用一种策略来解决这种冲突。采用对信息来源的偏好以及执行更高偏好的源的方式来解决冲突是一种普遍适用的方法。该论文提出了 “BoardgameQA” 数据集来测试语言模型的推理能力，结果表明当前最新技术的语言模型在处理相互矛盾的信息时存在显著差距。