Vertical Federated Learning (VFL) attracts increasing attention because it
empowers multiple parties to jointly train a privacy-preserving model over
vertically partitioned data. Recent research has shown that applying
zeroth-order optimization (ZOO) has many advantages in building a practical VFL
algorithm. However, a vital problem with the ZOO-based VFL is its slow
convergence rate, which limits its application in handling modern large models.
To address this problem, we propose a cascaded hybrid optimization method in
VFL. In this method, the downstream models (clients) are trained with ZOO to
protect privacy and ensure that no internal information is shared. Meanwhile,
the upstream model (server) is updated with first-order optimization (FOO)
locally, which significantly improves the convergence rate, making it feasible
to train the large models without compromising privacy and security. We
theoretically prove that our VFL framework converges faster than the ZOO-based
VFL, as the convergence of our framework is not limited by the size of the
server model, making it effective for training large models with the major part
on the server. Extensive experiments demonstrate that our method achieves
faster convergence than the ZOO-based VFL framework, while maintaining an
equivalent level of privacy protection. Moreover, we show that the convergence
of our VFL is comparable to the unsafe FOO-based VFL baseline. Additionally, we
demonstrate that our method makes the training of a large model feasible.

本文提出了一个级联混合优化方法来加速零阶优化算法，从而解决基于零阶优化的垂直联邦学习算法的收敛速度慢的问题。该算法在较大模型的训练中非常有效，且具有相当的隐私保护水平。