Fair graph learning plays a pivotal role in numerous practical applications. Recently, many fair graph learning methods have been proposed; however, their evaluation often relies on poorly constructed semi-synthetic datasets or substandard real-world datasets. In such cases, even a basic Multilayer Perceptron (MLP) can outperform Graph Neural Networks (GNNs) in both utility and fairness. In this work, we illustrate that many datasets fail to provide meaningful information in the edges, which may challenge the necessity of using graph structures in these problems. To address these issues, we develop and introduce a collection of synthetic, semi-synthetic, and real-world datasets that fulfill a broad spectrum of requirements. These datasets are thoughtfully designed to include relevant graph structures and bias information crucial for the fair evaluation of models. The proposed synthetic and semi-synthetic datasets offer the flexibility to create data with controllable bias parameters, thereby enabling the generation of desired datasets with user-defined bias values with ease. Moreover, we conduct systematic evaluations of these proposed datasets and establish a unified evaluation approach for fair graph learning models. Our extensive experimental results with fair graph learning methods across our datasets demonstrate their effectiveness in benchmarking the performance of these methods. Our datasets and the code for reproducing our experiments are available at https://github.com/XweiQ/Benchmark-GraphFairness.

公平图学习方法的评估常常依赖于构造不完善的半合成数据集或次标准的真实世界数据集。本研究针对这些问题，通过开发和引入一系列合成、半合成和真实世界数据集，旨在为公平模型的评估提供有意义的图结构和关键偏见信息。经过对提出的数据集的系统评估，为公平图学习模型建立了一种统一的评估方法。我们在数据集上对公平图学习方法进行了广泛的实验，证明了这些方法在性能评估方面的有效性。

解决公平图学习数据集的不足：走向新的基准