Deep learning models have demonstrated impressive performance in various
domains. However, the prolonged training time of these models remains a
critical problem. Manually designed parallel training strategies could enhance
efficiency but require considerable time and deliver little flexibility. Hence,
automatic parallelism is proposed to automate the parallel strategy searching
process. Even so, existing approaches suffer from sub-optimal strategy space
because they treat automatic parallelism as two independent stages, namely
inter- and intra-layer parallelism. To address this issue, we propose UniAP,
which utilizes mixed integer quadratic programming to unify inter- and
intra-layer automatic parallelism. To the best of our knowledge, UniAP is the
first work to unify these two categories to search for a globally optimal
strategy. The experimental results show that UniAP outperforms state-of-the-art
methods by up to 1.70$\times$ in throughput and reduces strategy searching time
by up to 16$\times$ across four Transformer-like models.

深度学习模型的训练时间长依然是一个关键问题，提出了自动并行化来解决这个问题，并提出了 UniAP 方法来统一内部和外部自动并行化，实验证明 UniAP 在吞吐量上超过了现有方法 1.70 倍，并且减少了搜索时间 16 倍。

UniAP: 统一的跨层和内层自动并行化方法：混合整数二次规划

UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed  Integer Quadratic Programming

Model parallelism has become necessary to train large neural networks.
However, finding a suitable model parallel schedule for an arbitrary neural
network is a non-trivial task due to the exploding search space. In this work,
we present a model parallelism framework TAP that automatically searches for
the best data and tensor parallel schedules. Leveraging the key insight that a
neural network can be represented as a directed acyclic graph, within which may
only exist a limited set of frequent subgraphs, we design a graph pruning
algorithm to fold the search space efficiently. TAP runs at sub-linear
complexity concerning the neural network size. Experiments show that TAP is
$20\times- 160\times$ faster than the state-of-the-art automatic parallelism
framework, and the performance of its discovered schedules is competitive with
the expert-engineered ones.

本文提出了一种模型并行框架 TAP，利用神经网络作为有向无环图的性质，设计了一种图剪枝算法以高效地搜索最佳的数据和张量并行计划，实验表明 TAP 比现有自动并行化框架快 20-160 倍，并且其发现的并行化计划比专家设计的计划具有竞争力。