Federated learning is a promising framework to train neural networks with widely distributed data. However, performance degrades heavily with heterogeneously distributed data. Recent work has shown this is due to the final layer of the network being most prone to local bias, some finding success freezing the final layer as an orthogonal classifier. We investigate the training dynamics of the classifier by applying SVD to the weights motivated by the observation that freezing weights results in constant singular values. We find that there are differences when training in IID and non-IID settings. Based on this finding, we introduce two regularization terms for local training to continuously emulate IID settings: (1) variance in the dimension-wise probability distribution of the classifier and (2) hyperspherical uniformity of representations of the encoder. These regularizations promote local models to act as if it were in an IID setting regardless of the local data distribution, thus offsetting proneness to bias while being flexible to the data. On extensive experiments in both label-shift and feature-shift settings, we verify that our method achieves highest performance by a large margin especially in highly non-IID cases in addition to being scalable to larger models and datasets.

对于分布式数据的异构性，冻结神经网络的最后一层作为正交分类器可以减轻局部偏差，在 IID 和非 IID 设置下，通过应用奇异值分解对权重进行分析，我们引入了两个正则化项，以连续地模拟 IID 设置，并通过广泛的实验验证了我们的方法在非 IID 情况下具有更高的性能。

FedUV：面向异构联邦学习的均匀性和方差