Distributed gradient descent (DGD) is an efficient way of implementing
gradient descent (GD), especially for large data sets, by dividing the
computation tasks into smaller subtasks and assigning to different computing
servers (CSs) to be executed in parallel. In standard parallel execution,
per-iteration waiting time is limited by the execution time of the straggling
servers. Coded DGD techniques have been introduced recently, which can tolerate
straggling servers via assigning redundant computation tasks to the CSs. In
most of the existing DGD schemes, either with coded computation or coded
communication, the non-straggling CSs transmit one message per iteration once
they complete all their assigned computation tasks. However, although the
straggling servers cannot complete all their assigned tasks, they are often
able to complete a certain portion of them. In this paper, we allow multiple
transmissions from each CS at each iteration in order to make sure a maximum
number of completed computations can be reported to the aggregating server
(AS), including the straggling servers. We numerically show that the average
completion time per iteration can be reduced significantly by slightly
increasing the communication load per server.

本文介绍了一种新的分布式梯度下降计算方法，通过引入冗余计算任务实现了容错，并允许每个计算服务器在每次迭代中发送多个信息以确保最大化完成任务数量， 从而显著降低了每次迭代的平均完成时间。