Distributed full-graph training of Graph Neural Networks (GNNs) over large
graphs is bandwidth-demanding and time-consuming. Frequent exchanges of node
features, embeddings and embedding gradients (all referred to as messages)
across devices bring significant communication overhead for nodes with remote
neighbors on other devices (marginal nodes) and unnecessary waiting time for
nodes without remote neighbors (central nodes) in the training graph. This
paper proposes an efficient GNN training system, AdaQP, to expedite distributed
full-graph GNN training. We stochastically quantize messages transferred across
devices to lower-precision integers for communication traffic reduction and
advocate communication-computation parallelization between marginal nodes and
central nodes. We provide theoretical analysis to prove fast training
convergence (at the rate of O(T^{-1}) with T being the total number of training
epochs) and design an adaptive quantization bit-width assignment scheme for
each message based on the analysis, targeting a good trade-off between training
convergence and efficiency. Extensive experiments on mainstream graph datasets
show that AdaQP substantially improves distributed full-graph training's
throughput (up to 3.01 X) with negligible accuracy drop (at most 0.30%) or even
accuracy improvement (up to 0.19%) in most cases, showing significant
advantages over the state-of-the-art works.

本文研究了分布式完全图训练的图神经网络（GNNs），提出了一种快速训练系统 AdaQP，并使用随机量化和通信计算并行化等技术来降低通信成本，实现了训练吞吐量的显著提升和误差微小的准确性改进。