Memory-based Temporal Graph Neural Networks (MTGNNs) are a class of temporal
graph neural networks that utilize a node memory module to capture and retain
long-term temporal dependencies, leading to superior performance compared to
memory-less counterparts. However, the iterative reading and updating process
of the memory module in MTGNNs to obtain up-to-date information needs to follow
the temporal dependencies. This introduces significant overhead and limits
training throughput. Existing optimizations for static GNNs are not directly
applicable to MTGNNs due to differences in training paradigm, model
architecture, and the absence of a memory module. Moreover, they do not
effectively address the challenges posed by temporal dependencies, making them
ineffective for MTGNN training. In this paper, we propose MSPipe, a general and
efficient framework for MTGNNs that maximizes training throughput while
maintaining model accuracy. Our design addresses the unique challenges
associated with fetching and updating node memory states in MTGNNs by
integrating staleness into the memory module. However, simply introducing a
predefined staleness bound in the memory module to break temporal dependencies
may lead to suboptimal performance and lack of generalizability across
different models and datasets. To solve this, we introduce an online pipeline
scheduling algorithm in MSPipe that strategically breaks temporal dependencies
with minimal staleness and delays memory fetching to obtain fresher memory
states. Moreover, we design a staleness mitigation mechanism to enhance
training convergence and model accuracy. We provide convergence analysis and
prove that MSPipe maintains the same convergence rate as vanilla sample-based
GNN training. Experimental results show that MSPipe achieves up to 2.45x
speed-up without sacrificing accuracy, making it a promising solution for
efficient MTGNN training.

提出了 MSPipe，一种提高 MTGNN 训练效率的通用且高效的框架，通过将陈旧性整合到内存模块中，通过在线管道调度算法可以在降低陈旧度的同时打破时间依赖，提高内存获取效率，增强训练稳定性和模型准确性。实验结果显示，MSPipe 在不损失准确性的情况下可以实现高达 2.45 倍的加速，成为有效的 MTGNN 训练解决方案。