Federated learning is a machine learning approach where multiple devices
collaboratively learn with the help of a parameter server by sharing only their
local updates. While gradient-based optimization techniques are widely adopted
in this domain, the curvature information that second-order methods exhibit is
crucial to guide and speed up the convergence. This paper introduces a scalable
second-order method, allowing the adoption of curvature information in
federated large models. Our method, coined Fed-Sophia, combines a weighted
moving average of the gradient with a clipping operation to find the descent
direction. In addition to that, a lightweight estimation of the Hessian's
diagonal is used to incorporate the curvature information. Numerical evaluation
shows the superiority, robustness, and scalability of the proposed Fed-Sophia
scheme compared to first and second-order baselines.

本文章介绍了一个可扩展的二阶方法 Fed-Sophia，通过结合梯度的加权移动平均和剪切操作来寻找下降方向，并使用海森矩阵对角线的轻量级估计来融入曲率信息。数值评估表明，与一阶和二阶基准相比，所提出的 Fed-Sophia 算法在优越性、鲁棒性和可扩展性方面表现出色。