There is growing research interest in measuring the statistical heterogeneity of clients' local datasets. Such measurements are used to estimate the suitability for collaborative training of personalized federated learning (PFL) models. Currently, these research endeavors are taking place in silos and there is a lack of a unified benchmark to provide a fair and convenient comparison among various approaches in common settings. We aim to bridge this important gap in this paper. The proposed benchmarking framework currently includes six representative approaches. Extensive experiments have been conducted to compare these approaches under five standard non-IID FL settings, providing much needed insights into which approaches are advantageous under which settings. The proposed framework offers useful guidance on the suitability of various data divergence measures in FL systems. It is beneficial for keeping related research activities on the right track in terms of: (1) designing PFL schemes, (2) selecting appropriate data heterogeneity evaluation approaches for specific FL application scenarios, and (3) addressing fairness issues in collaborative model training. The code is available at https://github.com/Xiaoni-61/DH-Benchmark.

本研究针对个性化联邦学习中客户端本地数据集的统计异质性测量缺乏统一基准的问题，提出了一个全面的基准测试框架，涵盖六种代表性方法。通过在五种标准非独立同分布的联邦学习设置下进行广泛实验，该框架为不同方法的适用性提供了宝贵的见解，从而指导个性化联邦学习方案的设计与数据评估方法的选择。

个性化联邦学习中数据异质性评估方法的基准测试