Large deep learning models have shown great potential for delivering
exceptional results in various applications. However, the training process can
be incredibly challenging due to the models' vast parameter sizes, often
consisting of hundreds of billions of parameters. Common distributed training
methods, such as data parallelism, tensor parallelism, and pipeline
parallelism, demand significant data communication throughout the process,
leading to prolonged wait times for some machines in physically distant
distributed systems. To address this issue, we propose a novel solution called
Hulk, which utilizes a modified graph neural network to optimize distributed
computing systems. Hulk not only optimizes data communication efficiency
between different countries or even different regions within the same city, but
also provides optimal distributed deployment of models in parallel. For
example, it can place certain layers on a machine in a specific region or pass
specific parameters of a model to a machine in a particular location. By using
Hulk in experiments, we were able to improve the time efficiency of training
large deep learning models on distributed systems by more than 20\%. Our open
source collection of unlabeled data:this https URL.

使用修改后的图神经网络，我们提出了一种名为 Hulk 的新颖解决方案，优化在分布式计算系统中训练大规模深度学习模型过程中的数据通信效率，可提高时间效率超过 20％。