Stochastic Gradient Descent is used for large datasets to train models to
reduce the training time. On top of that data parallelism is widely used as a
method to efficiently train neural networks using multiple worker nodes in
parallel. Synchronous and asynchronous approach to data parallelism is used by
most systems to train the model in parallel. However, both of them have their
drawbacks. We propose a third approach to data parallelism which is a hybrid
between synchronous and asynchronous approaches, using both approaches to train
the neural network. When the threshold function is selected appropriately to
gradually shift all parameter aggregation from asynchronous to synchronous, we
show that in a given time period our hybrid approach outperforms both
asynchronous and synchronous approaches.

我们提出了一种介于同步和异步方法之间的混合数据并行性方法，使用这两种方法训练神经网络，通过适当选择阈值函数来逐渐从异步转为同步的参数聚合，我们证明在一定时间范围内，我们的混合方法优于异步和同步方法。

并行随机梯度下降的混合方法

Hybrid Approach to Parallel Stochastic Gradient Descent

Distributed Stochastic Gradient Descent (SGD) when run in a synchronous
manner, suffers from delays in runtime as it waits for the slowest workers
(stragglers). Asynchronous methods can alleviate stragglers, but cause gradient
staleness that can adversely affect the convergence error. In this work, we
present a novel theoretical characterization of the speedup offered by
asynchronous methods by analyzing the trade-off between the error in the
trained model and the actual training runtime(wallclock time). The main novelty
in our work is that our runtime analysis considers random straggling delays,
which helps us design and compare distributed SGD algorithms that strike a
balance between straggling and staleness. We also provide a new error
convergence analysis of asynchronous SGD variants without bounded or
exponential delay assumptions. Finally, based on our theoretical
characterization of the error-runtime trade-off, we propose a method of
gradually varying synchronicity in distributed SGD and demonstrate its
performance on CIFAR10 dataset.

本研究分析同步和异步分布式随机梯度下降算法的误差和训练时间之间的权衡，考虑到随机拖延延迟，提出了逐渐变化同步性的方法，并在 CIFAR10 数据集上表现良好。

慢而稳定的梯度也能取得胜利

Slow and Stale Gradients Can Win the Race

We survey the recent distributed computing literature on checking whether a
given distributed system configuration satisfies a given boolean predicate,
i.e., whether the configuration is legal or illegal w.r.t. that predicate. We
consider classical distributed computing environments, including mostly
synchronous fault-free network computing (LOCAL and CONGEST models), but also
asynchronous crash-prone shared-memory computing (WAIT-FREE model), and mobile
computing (FSYNC model).

本研究调查了关于检查给定分布式系统配置是否符合给定布尔谓词的最近分布式计算文献，考虑了经典的分布式计算环境，包括大多数同步无故障网络计算（LOCAL 和 CONGEST 模型），以及异步崩溃共享内存计算（WAIT-FREE 模型）和移动计算（FSYNC 模型）。