Decentralized bilevel optimization has been actively studied in the past few
years since it has widespread applications in machine learning. However,
existing algorithms suffer from large communication complexity caused by the
estimation of stochastic hypergradient, limiting their application to
real-world tasks. To address this issue, we develop a novel decentralized
stochastic bilevel gradient descent algorithm under the heterogeneous setting,
which enjoys a small communication cost in each round and small communication
rounds. As such, it can achieve a much better communication complexity than
existing algorithms. Moreover, we extend our algorithm to the more challenging
decentralized multi-level optimization. To the best of our knowledge, this is
the first time achieving these theoretical results under the heterogeneous
setting. At last, the experimental results confirm the efficacy of our
algorithm.

我们开发了一种新颖的去中心化随机双层梯度下降算法，在异构环境中具有每轮小的通信成本和通信轮数，从而比现有算法实现了更好的通信复杂度。我们还将我们的算法扩展到更具挑战性的去中心化多级优化问题。根据我们的知识，我们是第一次在异构环境下实现这些理论结果。最后，实验结果证实了我们算法的有效性。