Federated learning refers to a distributed machine learning paradigm in which
data samples are decentralized and distributed among multiple clients. These
samples may exhibit statistical heterogeneity, which refers to data
distributions are not independent and identical across clients. Additionally,
system heterogeneity, or variations in the computational power of the clients,
introduces biases into federated learning. The combined effects of statistical
and system heterogeneity can significantly reduce the efficiency of federated
optimization. However, the impact of hybrid heterogeneity is not rigorously
discussed. This paper explores how hybrid heterogeneity affects federated
optimization by investigating server-side optimization. The theoretical results
indicate that adaptively maximizing gradient diversity in server update
direction can help mitigate the potential negative consequences of hybrid
heterogeneity. To this end, we introduce a novel server-side gradient-based
optimizer \textsc{FedAWARE} with theoretical guarantees provided. Intensive
experiments in heterogeneous federated settings demonstrate that our proposed
optimizer can significantly enhance the performance of federated learning
across varying degrees of hybrid heterogeneity.

研究了混合异构性如何影响联邦优化，通过调查服务器端优化表明，自适应地最大化梯度多样性在服务器更新方向上可以帮助缓解混合异构性的潜在负面影响，引入了具有理论保证的新型基于梯度的优化器 FedAWARE，通过在异构联邦设置中进行大量实验表明，我们的提议优化器可以显著提升不同程度的混合异构性下联邦学习的性能。

通过梯度多样性最大化解决联邦优化的混合异质性问题

Tackling Hybrid Heterogeneity on Federated Optimization via Gradient  Diversity Maximization

We present a theoretical study of server-side optimization in federated
learning. Our results are the first to show that the widely popular heuristic
of scaling the client updates with an extra parameter is very useful in the
context of Federated Averaging (FedAvg) with local passes over the client data.
Each local pass is performed without replacement using Random Reshuffling,
which is a key reason we can show improved complexities. In particular, we
prove that whenever the local stepsizes are small, and the update direction is
given by FedAvg in conjunction with Random Reshuffling over all clients, one
can take a big leap in the obtained direction and improve rates for convex,
strongly convex, and non-convex objectives. In particular, in non-convex regime
we get an enhancement of the rate of convergence from
$\mathcal{O}\left(\varepsilon^{-3}\right)$ to
$\mathcal{O}\left(\varepsilon^{-2}\right)$. This result is new even for Random
Reshuffling performed on a single node. In contrast, if the local stepsizes are
large, we prove that the noise of client sampling can be controlled by using a
small server-side stepsize. To the best of our knowledge, this is the first
time that local steps provably help to overcome the communication bottleneck.
Together, our results on the advantage of large and small server-side stepsizes
give a formal justification for the practice of adaptive server-side
optimization in federated learning. Moreover, we consider a variant of our
algorithm that supports partial client participation, which makes the method
more practical.

本文研究了在联邦学习中的服务端优化问题，运用随机重排等技术，证明在使用 Federated Averaging 算法的情况下，通过调整本地学习率，可以显著提高求解凸优化和非凸优化问题的效果。同时，通过选择合适的本地学习率，可以有效克服通信瓶颈问题。