Large language models (LLMs) have gained enormous attention from both
academia and industry, due to their exceptional ability in language generation
and extremely powerful generalization. However, current LLMs still output
unreliable content in practical reasoning tasks due to their inherent issues
(e.g., hallucination). To better disentangle this problem, in this paper, we
conduct an in-depth investigation to systematically explore the capability of
LLMs in logical reasoning. More in detail, we first investigate the deficiency
of LLMs in logical reasoning on different tasks, including event relation
extraction and deductive reasoning. Our study demonstrates that LLMs are not
good reasoners in solving tasks with rigorous reasoning and will produce
counterfactual answers, which require us to iteratively refine. Therefore, we
comprehensively explore different strategies to endow LLMs with logical
reasoning ability, and thus enable them to generate more logically consistent
answers across different scenarios. Based on our approach, we also contribute a
synthesized dataset (LLM-LR) involving multi-hop reasoning for evaluation and
pre-training. Extensive quantitative and qualitative analyses on different
tasks also validate the effectiveness and necessity of teaching LLMs with logic
and provide insights for solving practical tasks with LLMs in future work.

通过对大型语言模型的研究，我们发现它们在逻辑推理方面存在缺陷，导致其在任务解决中产生反事实的答案。为了解决这个问题，我们提出了多种策略，赋予大型语言模型逻辑推理能力，从而使其能够在不同场景中生成更符合逻辑的答案。我们还通过构建一个综合数据集 (LMM-LR) 对该方法进行了评估和预训练。在不同任务上进行了广泛的定量和定性分析，验证了通过逻辑训练大型语言模型的有效性和必要性，并为将来的工作提供了启示。