Federated Learning (FL) is an emerging collaborative machine learning
framework where multiple clients train the global model without sharing their
own datasets. In FL, the model inconsistency caused by the local data
heterogeneity across clients results in the near-orthogonality of client
updates, which leads to the global update norm reduction and slows down the
convergence. Most previous works focus on eliminating the difference of
parameters (or gradients) between the local and global models, which may fail
to reflect the model inconsistency due to the complex structure of the machine
learning model and the Euclidean space's limitation in meaningful geometric
representations. In this paper, we propose FedMRUR by adopting the manifold
model fusion scheme and a new global optimizer to alleviate the negative
impacts. Concretely, FedMRUR adopts a hyperbolic graph manifold regularizer
enforcing the representations of the data in the local and global models are
close to each other in a low-dimensional subspace. Because the machine learning
model has the graph structure, the distance in hyperbolic space can reflect the
model bias better than the Euclidean distance. In this way, FedMRUR exploits
the manifold structures of the representations to significantly reduce the
model inconsistency. FedMRUR also aggregates the client updates norms as the
global update norm, which can appropriately enlarge each client's contribution
to the global update, thereby mitigating the norm reduction introduced by the
near-orthogonality of client updates. Furthermore, we theoretically prove that
our algorithm can achieve a linear speedup property for non-convex setting
under partial client participation.Experiments demonstrate that FedMRUR can
achieve a new state-of-the-art (SOTA) accuracy with less communication.

我们提出了 FedMRUR 方法，采用流形模型融合方案和一种新的全局优化器来减轻模型不一致性的负面影响，并通过在低维子空间中使本地和全局模型的数据表示相互接近的双曲图流形正则化器，从而显著减少模型不一致性，实现了局部客户更新的正交性引入的规范缩减。我们还证明了该算法在非凸设置下可以实现线性加速性能，并通过实验证明 FedMRUR 在通信量较少的情况下可以实现新的最先进准确性。

具有流形正则化和归一化更新聚合的联邦学习

Federated Learning with Manifold Regularization and Normalized Update  Reaggregation

Asynchronous parallel optimization received substantial successes and
extensive attention recently. One of core theoretical questions is how much
speedup (or benefit) the asynchronous parallelization can bring us. This paper
provides a comprehensive and generic analysis to study the speedup property for
a broad range of asynchronous parallel stochastic algorithms from the zeroth
order to the first order methods. Our result recovers or improves existing
analysis on special cases, provides more insights for understanding the
asynchronous parallel behaviors, and suggests a novel asynchronous parallel
zeroth order method for the first time. Our experiments provide novel
applications including model blending problems using the proposed asynchronous
parallel zeroth order method.

本文研究异步并行优化问题，分析了从零阶到一阶方法的广泛异步并行随机算法的加速性能，提出了异步并行零阶方法并应用于模型混合问题。