Modern realities and trends in learning require more and more generalization ability of models, which leads to an increase in both models and training sample size. It is already difficult to solve such tasks in a single device mode. This is the reason why distributed and federated learning approaches are becoming more popular every day. Distributed computing involves communication between devices, which requires solving two key problems: efficiency and privacy. One of the most well-known approaches to combat communication costs is to exploit the similarity of local data. Both Hessian similarity and homogeneous gradients have been studied in the literature, but separately. In this paper, we combine both of these assumptions in analyzing a new method that incorporates the ideas of using data similarity and clients sampling. Moreover, to address privacy concerns, we apply the technique of additional noise and analyze its impact on the convergence of the proposed method. The theory is confirmed by training on real datasets.

本研究针对分布式和联邦学习中通信成本高和隐私保护的难题，提出了一种结合Hessian相似性和梯度相似性的新方法。该方法引入了数据相似性的思想以及客户端采样，同时通过噪声技术应对隐私问题。研究结果表明，所提方法能有效提升模型的收敛性，具有重要的实际应用潜力。

加速随机外梯度：混合 Hessian 和梯度相似性以减少分布式和联邦学习中的通信