Task-based distributed frameworks (e.g., Ray, Dask, Hydro) have become
increasingly popular for distributed applications that contain asynchronous and
dynamic workloads, including asynchronous gradient descent, reinforcement
learning, and model serving. As more data-intensive applications move to run on
top of task-based systems, collective communication efficiency has become an
important problem. Unfortunately, traditional collective communication
libraries (e.g., MPI, Horovod, NCCL) are an ill fit, because they require the
communication schedule to be known before runtime and they do not provide fault
tolerance.
We design and implement Hoplite, an efficient and fault-tolerant collective
communication layer for task-based distributed systems. Our key technique is to
compute data transfer schedules on the fly and execute the schedules
efficiently through fine-grained pipelining. At the same time, when a task
fails, the data transfer schedule adapts quickly to allow other tasks to keep
making progress. We apply Hoplite to a popular task-based distributed
framework, Ray. We show that Hoplite speeds up asynchronous stochastic gradient
descent, reinforcement learning, and serving an ensemble of machine learning
models that are difficult to execute efficiently with traditional collective
communication by up to 7.8x, 3.9x, and 3.3x, respectively.

Hoplite 是一种高效的、容错的、用于面向任务的分布式系统的集体通信层，它能够实时计算数据传输计划并通过细粒度流水线方式执行以处理异步和动态工作负载，同时提供容错能力，可应用于异步随机梯度下降等应用，提高传统集体通信效率达 7.8 倍。