Edge computing has recently emerged as a promising paradigm to boost the
performance of distributed learning by leveraging the distributed resources at
edge nodes. Architecturally, the introduction of edge nodes adds an additional
intermediate layer between the master and workers in the original distributed
learning systems, potentially leading to more severe straggler effect.
Recently, coding theory-based approaches have been proposed for stragglers
mitigation in distributed learning, but the majority focus on the conventional
workers-master architecture. In this paper, along a different line, we
investigate the problem of mitigating the straggler effect in hierarchical
distributed learning systems with an additional layer composed of edge nodes.
Technically, we first derive the fundamental trade-off between the
computational loads of workers and the stragglers tolerance. Then, we propose a
hierarchical gradient coding framework, which provides better stragglers
mitigation, to achieve the derived computational trade-off. To further improve
the performance of our framework in heterogeneous scenarios, we formulate an
optimization problem with the objective of minimizing the expected execution
time for each iteration in the learning process. We develop an efficient
algorithm to mathematically solve the problem by outputting the optimum
strategy. Extensive simulation results demonstrate the superiority of our
schemes compared with conventional solutions.

通过引入边缘节点作为额外的中间层，本文探讨了在分层分布式学习系统中缓解滞后效应的问题，首先推导了工作节点的计算负载和滞后者容忍度之间的基本权衡关系，然后提出了一种层级渐进编码框架，以实现所得到的计算权衡。为了在异构场景中进一步提高我们的框架性能，我们制定了一个优化问题，旨在最小化学习过程中每次迭代的预期执行时间，并设计了一种高效算法来数学求解该问题，得到最优策略。通过大量模拟结果，证明了我们的方案相对于传统解决方案的优越性。

基于边缘设备的分布式学习的分层梯度编码设计与优化

Design and Optimization of Hierarchical Gradient Coding for Distributed  Learning at Edge Devices

Distributed algorithms are often beset by the straggler effect, where the
slowest compute nodes in the system dictate the overall running time.
Coding-theoretic techniques have been recently proposed to mitigate stragglers
via algorithmic redundancy. Prior work in coded computation and gradient coding
has mainly focused on exact recovery of the desired output. However, slightly
inexact solutions can be acceptable in applications that are robust to noise,
such as model training via gradient-based algorithms. In this work, we present
computationally simple gradient codes based on sparse graphs that guarantee
fast and approximately accurate distributed computation. We demonstrate that
sacrificing a small amount of accuracy can significantly increase algorithmic
robustness to stragglers.

通过稀疏图构建的计算简单的渐变编码可以有效增强算法对滞后节点的鲁棒性。