Training Deep Learning (DL) models require large, high-quality datasets,
often assembled with data from different institutions. Federated Learning (FL)
has been emerging as a method for privacy-preserving pooling of datasets
employing collaborative training from different institutions by iteratively
globally aggregating locally trained models. One critical performance challenge
of FL is operating on datasets not independently and identically distributed
(non-IID) among the federation participants. Even though this fragility cannot
be eliminated, it can be debunked by a suitable optimization of two
hyper-parameters: layer normalization methods and collaboration frequency
selection. In this work, we benchmark five different normalization layers for
training Neural Networks (NNs), two families of non-IID data skew, and two
datasets. Results show that Batch Normalization, widely employed for
centralized DL, is not the best choice for FL, whereas Group and Layer
Normalization consistently outperform Batch Normalization. Similarly, frequent
model aggregation decreases convergence speed and mode quality.

本文研究了使用联邦学习方法在分布式非独立同分布数据集上训练深度学习模型时，采用不同的规范化层和协同频率策略，优化模型性能的效果，研究发现，Batch Normalization 并不是最适合联邦学习的规范化策略，而 Group 与 Layer Normalization 能更有效地提升模型的速度和质量。此外，频繁的模型聚合也能降低模型性能。