Evaluating the graph comprehension and reasoning abilities of Large Language Models (LLMs) is challenging and often incomplete. Existing benchmarks focus primarily on pure graph understanding, lacking a comprehensive evaluation across all graph types and detailed capability definitions. This paper presents GraCoRe, a benchmark for systematically assessing LLMs' graph comprehension and reasoning. GraCoRe uses a three-tier hierarchical taxonomy to categorize and test models on pure graph and heterogeneous graphs, subdividing capabilities into 10 distinct areas tested through 19 tasks. Our benchmark includes 11 datasets with 5,140 graphs of varying complexity. We evaluated three closed-source and seven open-source LLMs, conducting thorough analyses from both ability and task perspectives. Key findings reveal that semantic enrichment enhances reasoning performance, node ordering impacts task success, and the ability to process longer texts does not necessarily improve graph comprehension or reasoning. GraCoRe is open-sourced at https://github.com/ZIKEYUAN/GraCoRe

本文介绍了GraCoRe，这是一个用于系统评估LLMs的图形理解和推理能力的基准测试，它使用三层分层分类法对纯图形和异构图形进行测试，并通过19个任务对10个不同能力进行测试，研究发现语义增强提高了推理性能，节点排序影响任务成功，处理更长的文本并不一定提高图形理解或推理能力。

GraCoRe：大型语言模型中的图推理与复杂推理基准测试