Training Graph Neural Networks (GNNs) on large graphs presents unique
challenges due to the large memory and computing requirements. Distributed GNN
training, where the graph is partitioned across multiple machines, is a common
approach to training GNNs on large graphs. However, as the graph cannot
generally be decomposed into small non-interacting components, data
communication between the training machines quickly limits training speeds.
Compressing the communicated node activations by a fixed amount improves the
training speeds, but lowers the accuracy of the trained GNN. In this paper, we
introduce a variable compression scheme for reducing the communication volume
in distributed GNN training without compromising the accuracy of the learned
model. Based on our theoretical analysis, we derive a variable compression
method that converges to a solution equivalent to the full communication case,
for all graph partitioning schemes. Our empirical results show that our method
attains a comparable performance to the one obtained with full communication.
We outperform full communication at any fixed compression ratio for any
communication budget.

在分布式图神经网络训练中引入了一种变化的压缩方案，用于减少通信量而不降低学习模型的准确性，并通过理论分析和实证结果证明了其性能优于完全通信情况下的压缩比。

变通信速率下的大规模图神经网络分布式训练

Distributed Training of Large Graph Neural Networks with Variable  Communication Rates

We introduce a novel federated learning framework, FedD3, which reduces the
overall communication volume and with that opens up the concept of federated
learning to more application scenarios in network-constrained environments. It
achieves this by leveraging local dataset distillation instead of traditional
learning approaches (i) to significantly reduce communication volumes and (ii)
to limit transfers to one-shot communication, rather than iterative multiway
communication. Instead of sharing model updates, as in other federated learning
approaches, FedD3 allows the connected clients to distill the local datasets
independently, and then aggregates those decentralized distilled datasets
(typically in the form a few unrecognizable images, which are normally smaller
than a model) across the network only once to form the final model. Our
experimental results show that FedD3 significantly outperforms other federated
learning frameworks in terms of needed communication volumes, while it provides
the additional benefit to be able to balance the trade-off between accuracy and
communication cost, depending on usage scenario or target dataset. For
instance, for training an AlexNet model on a Non-IID CIFAR-10 dataset with 10
clients, FedD3 can either increase the accuracy by over 71% with a similar
communication volume, or save 98% of communication volume, while reaching the
same accuracy, comparing to other one-shot federated learning approaches.

介绍了一种新的联邦学习框架 FedD3，它通过利用本地数据集蒸馏来代替传统的学习方法显著减少通信量，仅在网络上集成分散的蒸馏数据集一次以形成最终模型，取得了比其他联邦学习框架显著的交流效益和平衡精度和通信成本之间的权衡的额外利益。