Large Language Models (LLMs) have achieved great success in various reasoning tasks. In this work, we focus on the graph reasoning ability of LLMs. Although theoretical studies proved that LLMs are capable of handling graph reasoning tasks, empirical evaluations reveal numerous failures. To deepen our understanding on this discrepancy, we revisit the ability of LLMs on three fundamental graph tasks: graph description translation, graph connectivity, and the shortest-path problem. Our findings suggest that LLMs can fail to understand graph structures through text descriptions and exhibit varying performance for all these three fundamental tasks. Meanwhile, we perform a real-world investigation on knowledge graphs and make consistent observations with our findings. The codes and datasets are available.

本研究针对大型语言模型在图推理任务中的表现进行分析，揭示了其在图结构理解方面的缺陷。通过深入探讨图描述翻译、图连通性和最短路径问题，研究结果表明语言模型在这些基本任务中存在显著的性能差异，尤其是在文本描述的图结构理解上表现不佳。

重新审视大型语言模型的图推理能力：翻译、连通性和最短路径案例研究