We perform a study of the factors affecting training time in multi-device deep learning systems. Given a specification of a convolutional neural network, we study how to minimize the time to train this model on a cluster of commodity CPUs and GPUs. Our first contribution focuses on the single-node setting, in which we show that by using standard batching and data-parallel techniques throughput can be improved by at least 5.5x over state-of-the-art systems when training on CPUs. This ensures an end-to-end training time directly proportional to the throughput of a device regardless of its underlying hardware, allowing each node in the cluster to be treated as a black box. Our second contribution is a theoretical and empirical study of the tradeoffs affecting end-to-end training time in a multiple-device setting. We identify the degree of asynchronous parallelization as a key feature affecting both hardware and statistical efficiency. We show that asynchrony can be viewed as introducing a momentum parameter, which we use to limit our search space; in turn, this leads to a simpler optimizer, which is our third contribution. Our optimizer involves a predictive model for the total time to convergence and selects an allocation of resources to minimize that time. We demonstrate that the most popular distributed deep learning systems fall within our tradeoff space but do not optimize within the space. By doing such optimization, our prototype runs 1.9x to 12x faster than the fastest state-of-the-art systems.

该研究通过标准批处理和数据并行技术，在单个节点环境中将吞吐量提高至少5.5倍，并在多节点环境中研究了异步并行化对训练时间的影响，并提出了一种高效的超参数优化器，以选择资源分配方式以最小化收敛所需的总时间。

Omnivore: 一种用于在CPU和GPU上进行多设备深度学习的优化器