Modern deep learning models, growing larger and more complex, have
demonstrated exceptional generalization and accuracy due to training on huge
datasets. This trend is expected to continue. However, the increasing size of
these models poses challenges in training, as traditional centralized methods
are limited by memory constraints at such scales. This paper proposes an
asynchronous decentralized training paradigm for large modern deep learning
models that harnesses the compute power of regular heterogeneous PCs with
limited resources connected across the internet to achieve favourable
performance metrics. Ravnest facilitates decentralized training by efficiently
organizing compute nodes into clusters with similar data transfer rates and
compute capabilities, without necessitating that each node hosts the entire
model. These clusters engage in $\textit{Zero-Bubble Asynchronous Model
Parallel}$ training, and a $\textit{Parallel Multi-Ring All-Reduce}$ method is
employed to effectively execute global parameter averaging across all clusters.
We have framed our asynchronous SGD loss function as a block structured
optimization problem with delayed updates and derived an optimal convergence
rate of $O\left(\frac{1}{\sqrt{K}}\right)$. We further discuss linear speedup
with respect to the number of participating clusters and the bound on the
staleness parameter.

该论文提出了一种异步分布式训练方法，利用互联网连接的具有有限资源的普通异构个人电脑的计算能力，在大型现代深度学习模型上取得了良好的性能指标，通过将计算节点高效地组织成具有相似数据传输速度和计算能力的集群，避免了每个节点承载整个模型，并采用并行多环全局参数平均方法实现全局参数平均。同时，论文在异步 SGD 损失函数上进行了解析，推导出了收敛速度的最优复杂度以及参与集群数量的线性加速和时滞参数的界.