Current distributed full-graph GNN training methods adopt a variant of data
parallelism, namely graph parallelism, in which the whole graph is divided into
multiple partitions (subgraphs) and each GPU processes one of them. This incurs
high communication overhead because of the inter-partition message passing at
each layer. To this end, we proposed a new training method named GNNPipe that
adopts model parallelism instead, which has a lower worst-case asymptotic
communication complexity than graph parallelism. To ensure high GPU
utilization, we proposed to combine model parallelism with a chunk-based
pipelined training method, in which each GPU processes a different chunk of
graph data at different layers concurrently. We further proposed hybrid
parallelism that combines model and graph parallelism when the model-level
parallelism is insufficient. We also introduced several tricks to ensure
convergence speed and model accuracies to accommodate embedding staleness
introduced by pipelining. Extensive experiments show that our method reduces
the per-epoch training time by up to 2.45x (on average 2.03x) and reduces the
communication volume and overhead by up to 22.51x and 27.21x (on average 10.27x
and 14.96x), respectively, while achieving a comparable level of model accuracy
and convergence speed compared to graph parallelism.

采用模型并行而非图并行的分布式全局图神经网络训练方法 GNNPipe，结合基于分块的流水线训练方法以及混合并行性，以减少通信开销并加快训练时间，同时保持相当的模型准确性和收敛速度。

GNNPipe: 使用流水线模型并行加速分布式全图 GNN 训练

GNNPipe: Accelerating Distributed Full-Graph GNN Training with Pipelined  Model Parallelism

Recent progress in Graph Neural Networks (GNNs) for modeling atomic
simulations has the potential to revolutionize catalyst discovery, which is a
key step in making progress towards the energy breakthroughs needed to combat
climate change. However, the GNNs that have proven most effective for this task
are memory intensive as they model higher-order interactions in the graphs such
as those between triplets or quadruplets of atoms, making it challenging to
scale these models. In this paper, we introduce Graph Parallelism, a method to
distribute input graphs across multiple GPUs, enabling us to train very large
GNNs with hundreds of millions or billions of parameters. We empirically
evaluate our method by scaling up the number of parameters of the recently
proposed DimeNet++ and GemNet models by over an order of magnitude. On the
large-scale Open Catalyst 2020 (OC20) dataset, these graph-parallelized models
lead to relative improvements of 1) 15% on the force MAE metric for the S2EF
task and 2) 21% on the AFbT metric for the IS2RS task, establishing new
state-of-the-art results.

本文提出的图并行算法可在多个 GPU 上分布输入图，从而实现训练数十亿个参数的大型 GNN， 该方法在大规模开放铂族催化剂的数据集上取得了新的最先进结果。