We present distributed algorithms for training dynamic Graph Neural Networks
(GNN) on large scale graphs spanning multi-node, multi-GPU systems. To the best
of our knowledge, this is the first scaling study on dynamic GNN. We devise
mechanisms for reducing the GPU memory usage and identify two execution time
bottlenecks: CPU-GPU data transfer; and communication volume. Exploiting
properties of dynamic graphs, we design a graph difference-based strategy to
significantly reduce the transfer time. We develop a simple, but effective data
distribution technique under which the communication volume remains fixed and
linear in the input size, for any number of GPUs. Our experiments using
billion-size graphs on a system of 128 GPUs shows that: (i) the distribution
scheme achieves up to 30x speedup on 128 GPUs; (ii) the graph-difference
technique reduces the transfer time by a factor of up to 4.1x and the overall
execution time by up to 40%

本文介绍了一种在大规模动态图上训练分布式算法的方法，采用图差分策略和数据分布技术极大地降低了传输和运行时间，并在使用 128 个 GPU 的系统上取得了高达 30 倍的加速。