Despite significant advancements, there is a limited understanding of how
large language models (LLMs) utilize knowledge for reasoning. To address this,
we propose a method that deconstructs complex real-world questions into a
graph, representing each question as a node with parent nodes of background
knowledge needed to solve the question. We develop the DepthQA dataset,
deconstructing questions into three depths: (i) recalling conceptual knowledge,
(ii) applying procedural knowledge, and (iii) analyzing strategic knowledge.
Based on a hierarchical graph, we quantify forward discrepancy, discrepancies
in LLMs' performance on simpler sub-problems versus complex questions. We also
measure backward discrepancy, where LLMs answer complex questions but struggle
with simpler ones. Our analysis shows that smaller models have more
discrepancies than larger models. Additionally, guiding models from simpler to
complex questions through multi-turn interactions improves performance across
model sizes, highlighting the importance of structured intermediate steps in
knowledge reasoning. This work enhances our understanding of LLM reasoning and
suggests ways to improve their problem-solving abilities.

通过将复杂的现实世界问题分解成图形，将每个问题表示为一个节点，并使用具有解决问题所需背景知识的父节点来研究大型语言模型（LLMs）如何利用知识进行推理。使用分层图形，我们量化了 LLMs 在较简单子问题与复杂问题上性能的前向差异和后向差异。此研究拓展了我们对 LLM 推理的理解，并提出了改进它们解决问题能力的方法。