Training Graph Neural Networks (GNNs) on large graphs is challenging due to
the conflict between the high memory demand and limited GPU memory. Recently,
distributed full-graph GNN training has been widely adopted to tackle this
problem. However, the substantial inter-GPU communication overhead can cause
severe throughput degradation. Existing communication compression techniques
mainly focus on traditional DNN training, whose bottleneck lies in
synchronizing gradients and parameters. We find they do not work well in
distributed GNN training as the barrier is the layer-wise communication of
features during the forward pass & feature gradients during the backward pass.
To this end, we propose an efficient distributed GNN training framework Sylvie,
which employs one-bit quantization technique in GNNs and further pipelines the
curtailed communication with computation to enormously shrink the overhead
while maintaining the model quality. In detail, Sylvie provides a lightweight
Low-bit Module to quantize the sent data and dequantize the received data back
to full precision values in each layer. Additionally, we propose a Bounded
Staleness Adaptor to control the introduced staleness to achieve further
performance enhancement. We conduct theoretical convergence analysis and
extensive experiments on various models & datasets to demonstrate Sylvie can
considerably boost the training throughput by up to 28.1x.

本文提出了一种高效的分布式图神经网络（GNN）训练框架 Sylvie，该框架采用一位量化技术和有界延迟的调整器来缩减通信开销，同时保持模型质量，在各种模型和数据集上进行了广泛实验验证，结果表明，该框架能够将训练吞吐量显著提高 28.1 倍。

使用异步一比特通信提升分布式全图 GNN 训练

Boosting Distributed Full-graph GNN Training with Asynchronous One-bit Communication

Federated learning (FL) is a promising paradigm that enables collaboratively
learning a shared model across massive clients while keeping the training data
locally. However, for many existing FL systems, clients need to frequently
exchange model parameters of large data size with the remote cloud server
directly via wide-area networks (WAN), leading to significant communication
overhead and long transmission time. To mitigate the communication bottleneck,
we resort to the hierarchical federated learning paradigm of HiFL, which reaps
the benefits of mobile edge computing and combines synchronous client-edge
model aggregation and asynchronous edge-cloud model aggregation together to
greatly reduce the traffic volumes of WAN transmissions. Specifically, we first
analyze the convergence bound of HiFL theoretically and identify the key
controllable factors for model performance improvement. We then advocate an
enhanced design of HiFlash by innovatively integrating deep reinforcement
learning based adaptive staleness control and heterogeneity-aware client-edge
association strategy to boost the system efficiency and mitigate the staleness
effect without compromising model accuracy. Extensive experiments corroborate
the superior performance of HiFlash in model accuracy, communication reduction,
and system efficiency.

利用 hierarchical federated learning paradigm (HiFL)，基于手机边缘计算和同步客户端 - 边缘模型聚合与异步边缘 - 云模型聚合相结合的方式，显著降低了 WAN 传输的流量，提高了 HiFlash 的系统性能和精度。