Decentralized bilevel optimization has been actively studied in the past few
years since it has widespread applications in machine learning. However,
existing algorithms suffer from large communication complexity caused by the
estimation of stochastic hypergradient, limiting their application to
real-world tasks. To address this issue, we develop a novel decentralized
stochastic bilevel gradient descent algorithm under the heterogeneous setting,
which enjoys a small communication cost in each round and small communication
rounds. As such, it can achieve a much better communication complexity than
existing algorithms. Moreover, we extend our algorithm to the more challenging
decentralized multi-level optimization. To the best of our knowledge, this is
the first time achieving these theoretical results under the heterogeneous
setting. At last, the experimental results confirm the efficacy of our
algorithm.

我们开发了一种新颖的去中心化随机双层梯度下降算法，在异构环境中具有每轮小的通信成本和通信轮数，从而比现有算法实现了更好的通信复杂度。我们还将我们的算法扩展到更具挑战性的去中心化多级优化问题。根据我们的知识，我们是第一次在异构环境下实现这些理论结果。最后，实验结果证实了我们算法的有效性。

分散双层优化的通信复杂度

On the Communication Complexity of Decentralized Bilevel Optimization

We analyze asynchronous-type algorithms for distributed SGD in the
heterogeneous setting, where each worker has its own computation and
communication speeds, as well as data distribution. In these algorithms,
workers compute possibly stale and stochastic gradients associated with their
local data at some iteration back in history and then return those gradients to
the server without synchronizing with other workers. We present a unified
convergence theory for non-convex smooth functions in the heterogeneous regime.
The proposed analysis provides convergence for pure asynchronous SGD and its
various modifications. Moreover, our theory explains what affects the
convergence rate and what can be done to improve the performance of
asynchronous algorithms. In particular, we introduce a novel asynchronous
method based on worker shuffling. As a by-product of our analysis, we also
demonstrate convergence guarantees for gradient-type algorithms such as SGD
with random reshuffling and shuffle-once mini-batch SGD. The derived rates
match the best-known results for those algorithms, highlighting the tightness
of our approach. Finally, our numerical evaluations support theoretical
findings and show the good practical performance of our method.

异步类型算法在异构设置中被分析，提出了统一的收敛理论，并介绍了基于工人洗牌的新型异步方法，数值评估支持理论发现并展示了该方法的良好实际性能。

AsGrad: 异步 SGD 算法的锐利统一分析

AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithms

The angular synchronization problem aims to accurately estimate (up to a
constant additive phase) a set of unknown angles $\theta_1, \dots,
\theta_n\in[0, 2\pi)$ from $m$ noisy measurements of their offsets
$\theta_i-\theta_j \;\mbox{mod} \; 2\pi.$ Applications include, for example,
sensor network localization, phase retrieval, and distributed clock
synchronization. An extension of the problem to the heterogeneous setting
(dubbed $k$-synchronization) is to estimate $k$ groups of angles
simultaneously, given noisy observations (with unknown group assignment) from
each group. Existing methods for angular synchronization usually perform poorly
in high-noise regimes, which are common in applications. In this paper, we
leverage neural networks for the angular synchronization problem, and its
heterogeneous extension, by proposing GNNSync, a theoretically-grounded
end-to-end trainable framework using directed graph neural networks. In
addition, new loss functions are devised to encode synchronization objectives.
Experimental results on extensive data sets demonstrate that GNNSync attains
competitive, and often superior, performance against a comprehensive set of
baselines for the angular synchronization problem and its extension, validating
the robustness of GNNSync even at high noise levels.

使用有向图神经网络提出了 GNNSync，一个理论基础的端到端训练框架，用于解决角度同步问题及其异构扩展，并通过新的损失函数在广泛数据集上进行实验证明 GNNSync 在高噪声水平下具有竞争力和优越性能。