As Graph Neural Networks (GNNs) increase in popularity for scientific machine
learning, their training and inference efficiency is becoming increasingly
critical. Additionally, the deep learning field as a whole is trending towards
wider and deeper networks, and ever increasing data sizes, to the point where
hard hardware bottlenecks are often encountered. Emerging specialty hardware
platforms provide an exciting solution to this problem. In this paper, we
systematically profile and select low-level operations pertinent to GNNs for
scientific computing implemented in the Pytorch Geometric software framework.
These are then rigorously benchmarked on NVIDIA A100 GPUs for several various
combinations of input values, including tensor sparsity. We then analyze these
results for each operation. At a high level, we conclude that on NVIDIA
systems: (1) confounding bottlenecks such as memory inefficiency often dominate
runtime costs moreso than data sparsity alone, (2) native Pytorch operations
are often as or more competitive than their Pytorch Geometric equivalents,
especially at low to moderate levels of input data sparsity, and (3) many
operations central to state-of-the-art GNN architectures have little to no
optimization for sparsity. We hope that these results serve as a baseline for
those developing these operations on specialized hardware and that our
subsequent analysis helps to facilitate future software and hardware based
optimizations of these operations and thus scalable GNN performance as a whole.

通过在 PyTorch Geometric 软件框架中实现科学计算所需的低级操作并在 NVIDIA A100 GPU 上进行基准测试，我们发现在专用硬件平台上具有关键作用的操作时限制内存效率的瓶颈而不仅仅是数据稀疏性，希望这些结果能成为此类操作的基准，并有助于未来的软件和硬件优化，从而实现可扩展的 GNN 性能的整体增强。