The computational requirements for training deep neural networks (DNNs) have
grown to the point that it is now standard practice to parallelize training.
Existing deep learning systems commonly use data or model parallelism, but
unfortunately, these strategies often result in suboptimal parallelization
performance.
In this paper, we define a more comprehensive search space of parallelization
strategies for DNNs called SOAP, which includes strategies to parallelize a DNN
in the Sample, Operation, Attribute, and Parameter dimensions. We also propose
FlexFlow, a deep learning framework that uses guided randomized search of the
SOAP space to find a fast parallelization strategy for a specific parallel
machine. To accelerate this search, FlexFlow introduces a novel execution
simulator that can accurately predict a parallelization strategy's performance
and is three orders of magnitude faster than prior approaches that have to
execute each strategy. We evaluate FlexFlow with six real-world DNN benchmarks
on two GPU clusters and show that FlexFlow can increase training throughput by
up to 3.8x over state-of-the-art approaches, even when including its search
time, and also improves scalability.

本文提出了一个名为 SOAP 的深度神经网络并行化搜索空间，并使用指导性随机搜索来发现特定并行计算机的快速并行化策略，并提出了一种新型执行模拟器，可精确预测并行化策略的性能，可将训练吞吐量提高多达 3.8 倍，并提高了可扩展性。