Federated learning is a distributed learning framework which enables clients to train models individually and to upload their model updates for aggregation. The local training process heavily relies on distributed gradient descent techniques. In the situation where gradient information is not available, the gradients need to be estimated from zeroth-order information, which typically involves computing finite-differences along isotropic random directions. This method suffers from high estimation errors, as the geometric features of the objective landscape may be overlooked during the isotropic sampling. In this work, we propose a non-isotropic sampling method to improve the gradient estimation procedure. Gradients in our method are estimated in a subspace spanned by historical trajectories of solutions, aiming to encourage the exploration of promising regions and hence improve the convergence. We implement this method in zeroth-order federated settings, and show that the convergence rate aligns with existing ones while introducing no significant overheads in communication or local computation. The effectiveness of our proposal is verified on several numerical experiments in comparison to several commonly-used zeroth-order federated optimization algorithms.

本研究解决了在缺乏梯度信息的情况下，当前零阶优化方法中估计错误较高的问题。通过提出一种非各向同性采样方法，以历史解轨迹作为子空间进行梯度估计，从而改善收敛性。研究结果表明，该方法在联邦设置中的收敛速度与现有方法相当，并且在通信和本地计算上没有显著的额外开销。

历史轨迹辅助的零阶联邦优化