Federated Learning (FL) is an emerging collaborative machine learning
framework where multiple clients train the global model without sharing their
own datasets. In FL, the model inconsistency caused by the local data
heterogeneity across clients results in the near-orthogonality of client
updates, which leads to the global update norm reduction and slows down the
convergence. Most previous works focus on eliminating the difference of
parameters (or gradients) between the local and global models, which may fail
to reflect the model inconsistency due to the complex structure of the machine
learning model and the Euclidean space's limitation in meaningful geometric
representations. In this paper, we propose FedMRUR by adopting the manifold
model fusion scheme and a new global optimizer to alleviate the negative
impacts. Concretely, FedMRUR adopts a hyperbolic graph manifold regularizer
enforcing the representations of the data in the local and global models are
close to each other in a low-dimensional subspace. Because the machine learning
model has the graph structure, the distance in hyperbolic space can reflect the
model bias better than the Euclidean distance. In this way, FedMRUR exploits
the manifold structures of the representations to significantly reduce the
model inconsistency. FedMRUR also aggregates the client updates norms as the
global update norm, which can appropriately enlarge each client's contribution
to the global update, thereby mitigating the norm reduction introduced by the
near-orthogonality of client updates. Furthermore, we theoretically prove that
our algorithm can achieve a linear speedup property for non-convex setting
under partial client participation.Experiments demonstrate that FedMRUR can
achieve a new state-of-the-art (SOTA) accuracy with less communication.

我们提出了 FedMRUR 方法，采用流形模型融合方案和一种新的全局优化器来减轻模型不一致性的负面影响，并通过在低维子空间中使本地和全局模型的数据表示相互接近的双曲图流形正则化器，从而显著减少模型不一致性，实现了局部客户更新的正交性引入的规范缩减。我们还证明了该算法在非凸设置下可以实现线性加速性能，并通过实验证明 FedMRUR 在通信量较少的情况下可以实现新的最先进准确性。