Deep learning models have demonstrated impressive performance in various
domains. However, the prolonged training time of these models remains a
critical problem. Manually designed parallel training strategies could enhance
efficiency but require considerable time and deliver little flexibility. Hence,
automatic parallelism is proposed to automate the parallel strategy searching
process. Even so, existing approaches suffer from sub-optimal strategy space
because they treat automatic parallelism as two independent stages, namely
inter- and intra-layer parallelism. To address this issue, we propose UniAP,
which utilizes mixed integer quadratic programming to unify inter- and
intra-layer automatic parallelism. To the best of our knowledge, UniAP is the
first work to unify these two categories to search for a globally optimal
strategy. The experimental results show that UniAP outperforms state-of-the-art
methods by up to 1.70$\times$ in throughput and reduces strategy searching time
by up to 16$\times$ across four Transformer-like models.

深度学习模型的训练时间长依然是一个关键问题，提出了自动并行化来解决这个问题，并提出了 UniAP 方法来统一内部和外部自动并行化，实验证明 UniAP 在吞吐量上超过了现有方法 1.70 倍，并且减少了搜索时间 16 倍。