Federated Learning is increasingly used in domains such as healthcare to
facilitate collaborative model training without data-sharing. However, datasets
located in different sites are often non-identically distributed, leading to
degradation of model performance in FL. Most existing methods for assessing
these distribution shifts are limited by being dataset or task-specific.
Moreover, these metrics can only be calculated by exchanging data, a practice
restricted in many FL scenarios. To address these challenges, we propose a
novel metric for assessing dataset similarity. Our metric exhibits several
desirable properties for FL: it is dataset-agnostic, is calculated in a
privacy-preserving manner, and is computationally efficient, requiring no model
training. In this paper, we first establish a theoretical connection between
our metric and training dynamics in FL. Next, we extensively evaluate our
metric on a range of datasets including synthetic, benchmark, and medical
imaging datasets. We demonstrate that our metric shows a robust and
interpretable relationship with model performance and can be calculated in
privacy-preserving manner. As the first federated dataset similarity metric, we
believe this metric can better facilitate successful collaborations between
sites.

我们提出了一种用于评估数据集相似性的新型指标，该指标具有 FL 所需的几个理想特性：与数据集无关、以隐私保护的方式计算、计算效率高，无需模型训练。我们在合成、基准和医疗成像数据集等一系列数据集上对这一指标进行了广泛评估，证明了它与模型性能具有稳健且可解释的关系，并且可以以隐私保护的方式计算。作为第一个联邦数据集相似性指标，我们相信这一指标可以更好地促进站点之间的成功合作。

跨域联邦学习的数据集相似度通用度量

A Universal Metric of Dataset Similarity for Cross-silo Federated  Learning

In this paper we present an exploratory research on quantifying the impact
that data distribution has on the performance and evaluation of NLP models. We
propose an automated framework that measures the data point distribution across
6 different dimensions: ambiguity, difficulty, discriminability, length, noise,
and perplexity.
We use disproportional stratified sampling to measure how much the data
distribution affects absolute (Acc/F1) and relative (Rank) model performance.
We experiment on 2 different datasets (SQUAD and MNLI) and test a total of 135
different models (125 on SQUAD and 10 on MNLI). We demonstrate that without
explicit control of the data distribution, standard evaluation frameworks are
inconsistent and unreliable. We find that the impact of the data is
statistically significant and is often larger than the impact of changing the
metric.
In a second set of experiments, we demonstrate that the impact of data on
evaluation is not just observable, but also predictable. We propose to use
benchmark transparency as a method for comparing datasets and quantifying the
similarity between them. We find that the ``dataset similarity vector'' can be
used to predict how well a model generalizes out of distribution.

本文提出了一个自动化框架来测量数据分布对自然语言处理模型性能和评估的影响，并通过两组实验表明数据的分布对评估具有统计学上的重要性，并且可预测模型的泛化能力。

基准透明度：测量数据对评估的影响

Benchmark Transparency: Measuring the Impact of Data on Evaluation

Transfer learning is a deep-learning technique that ameliorates the problem
of learning when human-annotated labels are expensive and limited. In place of
such labels, it uses instead the previously trained weights from a well-chosen
source model as the initial weights for the training of a base model for a new
target dataset. We demonstrate a novel but general technique for automatically
creating such source models. We generate pseudo-labels according to an
efficient and extensible algorithm that is based on a classical result from the
geometry of high dimensions, the Cayley-Menger determinant. This G2L
(``geometry to label'') method incrementally builds up pseudo-labels using a
greedy computation of hypervolume content. We demonstrate that the method is
tunable with respect to expected accuracy, which can be forecast by an
information-theoretic measure of dataset similarity (divergence) between source
and target. The results of 280 experiments show that this mechanical technique
generates base models that have similar or better transferability compared to a
baseline of models trained on extensively human-annotated ImageNet1K labels,
yielding an overall error decrease of 0.43\%, and an error decrease in 4 out of
5 divergent datasets tested.

采用几何学算法 Cayley-Menger 行列式的 C2L 方法生成伪标签，以在转移学习中自动创建源模型，与经过广泛人类注释的 ImageNet1K 标签相比，其基本模型具有相似或更好的可转移性，从而使最终错误率降低 0.43％，其中 5 个不同数据集中的 4 个错误率减少。