Current distributed full-graph GNN training methods adopt a variant of data
parallelism, namely graph parallelism, in which the whole graph is divided into
multiple partitions (subgraphs) and each GPU processes one of them. This incurs
high communication overhead because of the inter-partition message passing at
each layer. To this end, we proposed a new training method named GNNPipe that
adopts model parallelism instead, which has a lower worst-case asymptotic
communication complexity than graph parallelism. To ensure high GPU
utilization, we proposed to combine model parallelism with a chunk-based
pipelined training method, in which each GPU processes a different chunk of
graph data at different layers concurrently. We further proposed hybrid
parallelism that combines model and graph parallelism when the model-level
parallelism is insufficient. We also introduced several tricks to ensure
convergence speed and model accuracies to accommodate embedding staleness
introduced by pipelining. Extensive experiments show that our method reduces
the per-epoch training time by up to 2.45x (on average 2.03x) and reduces the
communication volume and overhead by up to 22.51x and 27.21x (on average 10.27x
and 14.96x), respectively, while achieving a comparable level of model accuracy
and convergence speed compared to graph parallelism.

采用模型并行而非图并行的分布式全局图神经网络训练方法 GNNPipe，结合基于分块的流水线训练方法以及混合并行性，以减少通信开销并加快训练时间，同时保持相当的模型准确性和收敛速度。

GNNPipe: 使用流水线模型并行加速分布式全图 GNN 训练

GNNPipe: Accelerating Distributed Full-Graph GNN Training with Pipelined  Model Parallelism

In this paper, we consider hybrid parallelism -- a paradigm that employs both
Data Parallelism (DP) and Model Parallelism (MP) -- to scale distributed
training of large recommendation models. We propose a compression framework
called Dynamic Communication Thresholding (DCT) for communication-efficient
hybrid training. DCT filters the entities to be communicated across the network
through a simple hard-thresholding function, allowing only the most relevant
information to pass through. For communication efficient DP, DCT compresses the
parameter gradients sent to the parameter server during model synchronization.
The threshold is updated only once every few thousand iterations to reduce the
computational overhead of compression. For communication efficient MP, DCT
incorporates a novel technique to compress the activations and gradients sent
across the network during the forward and backward propagation, respectively.
This is done by identifying and updating only the most relevant neurons of the
neural network for each training sample in the data. We evaluate DCT on
publicly available natural language processing and recommender models and
datasets, as well as recommendation systems used in production at Facebook. DCT
reduces communication by at least $100\times$ and $20\times$ during DP and MP,
respectively. The algorithm has been deployed in production, and it improves
end-to-end training time for a state-of-the-art industrial recommender model by
37\%, without any loss in performance.

本文针对分布式系统中大规模推荐模型的分布式训练提出了一种基于数据并行性和模型并行性的混合并行计算模型，并设计了动态通信阈值压缩框架，该框架能够有效地减少通信开销并提高训练效率。实验结果表明，使用该算法可以在不降低性能的情况下，将数据并行和模型并行的通信量分别降低至原来的 1% 和 5%。

大规模培训推荐系统：通信高效模型和数据并行处理

Training Recommender Systems at Scale: Communication-Efficient Model and  Data Parallelism

Nowadays, deep neural networks (DNNs) are the core enablers for many emerging
edge AI applications. Conventional approaches to training DNNs are generally
implemented at central servers or cloud centers for centralized learning, which
is typically time-consuming and resource-demanding due to the transmission of a
large amount of data samples from the device to the remote cloud. To overcome
these disadvantages, we consider accelerating the learning process of DNNs on
the Mobile-Edge-Cloud Computing (MECC) paradigm. In this paper, we propose
HierTrain, a hierarchical edge AI learning framework, which efficiently deploys
the DNN training task over the hierarchical MECC architecture. We develop a
novel \textit{hybrid parallelism} method, which is the key to HierTrain, to
adaptively assign the DNN model layers and the data samples across the three
levels of edge device, edge server and cloud center. We then formulate the
problem of scheduling the DNN training tasks at both layer-granularity and
sample-granularity. Solving this optimization problem enables us to achieve the
minimum training time. We further implement a hardware prototype consisting of
an edge device, an edge server and a cloud server, and conduct extensive
experiments on it. Experimental results demonstrate that HierTrain can achieve
up to 6.9x speedup compared to the cloud-based hierarchical training approach.

提出了一种名为 HierTrain 的层次化边缘 AI 学习框架，使用深度神经网络在移动边缘云计算中进行训练，采用了新型的混合并行方法，旨在在边缘设备、边缘服务器和云中进行 DNN 模型层次的自适应分配，该框架在硬件原型上实现，比基于云的 Hierarchical Training 方法快了多达 6.9 倍。