With the rapid proliferation of scientific literature, versatile academic
knowledge services increasingly rely on comprehensive academic graph mining.
Despite the availability of public academic graphs, benchmarks, and datasets,
these resources often fall short in multi-aspect and fine-grained annotations,
are constrained to specific task types and domains, or lack underlying real
academic graphs. In this paper, we present OAG-Bench, a comprehensive,
multi-aspect, and fine-grained human-curated benchmark based on the Open
Academic Graph (OAG). OAG-Bench covers 10 tasks, 20 datasets, 70+ baselines,
and 120+ experimental results to date. We propose new data annotation
strategies for certain tasks and offer a suite of data pre-processing codes,
algorithm implementations, and standardized evaluation protocols to facilitate
academic graph mining. Extensive experiments reveal that even advanced
algorithms like large language models (LLMs) encounter difficulties in
addressing key challenges in certain tasks, such as paper source tracing and
scholar profiling. We also introduce the Open Academic Graph Challenge
(OAG-Challenge) to encourage community input and sharing. We envisage that
OAG-Bench can serve as a common ground for the community to evaluate and
compare algorithms in academic graph mining, thereby accelerating algorithm
development and advancement in this field. OAG-Bench is accessible at
this https URL

本文介绍了基于开放学术图的全面、多方面和细粒度的人工筛选基准 OAG-Bench，涵盖了 10 个任务、20 个数据集、70 + 基线和 120 + 实验结果，并提供了新的数据注释策略、数据预处理代码、算法实现和标准化评估协议，以促进学术图挖掘的发展。

OAG-Bench: 学术图挖掘的人工策划基准

OAG-Bench: A Human-Curated Benchmark for Academic Graph Mining

Academic knowledge services have substantially facilitated the development of
the science enterprise by providing a plenitude of efficient research tools.
However, many applications highly depend on ad-hoc models and expensive human
labeling to understand scientific contents, hindering deployments into real
products. To build a unified backbone language model for different
knowledge-intensive academic applications, we pre-train an academic language
model OAG-BERT that integrates both the heterogeneous entity knowledge and
scientific corpora in the Open Academic Graph (OAG) -- the largest public
academic graph to date. In OAG-BERT, we develop strategies for pre-training
text and entity data along with zero-shot inference techniques. In OAG-BERT, we
develop strategies for pre-training text and entity data along with zero-shot
inference techniques. Its zero-shot capability furthers the path to mitigate
the need of expensive annotations. OAG-BERT has been deployed for real-world
applications, such as the reviewer recommendation function for National Nature
Science Foundation of China (NSFC) -- one of the largest funding agencies in
China -- and paper tagging in AMiner. All codes and pre-trained models are
available via the CogDL toolkit.

通过使用 Open Academic Graph 中的异构实体知识和科学语料库，我们为不同的高度知识密集型的学术应用程序构建了一个统一的骨干语言模型 OAG-BERT，并开发了其预训练策略和零 - shot 推理技术，并将其应用于实际应用程序中。