Efficient training of large-scale graph neural networks (GNNs) has been
studied with a specific focus on reducing their memory consumption. Work by Liu
et al. (2022) proposed extreme activation compression (EXACT) which
demonstrated drastic reduction in memory consumption by performing quantization
of the intermediate activation maps down to using INT2 precision. They showed
little to no reduction in performance while achieving large reductions in GPU
memory consumption. In this work, we present an improvement to the EXACT
strategy by using block-wise quantization of the intermediate activation maps.
We experimentally analyze different block sizes and show further reduction in
memory consumption (>15%), and runtime speedup per epoch (about 5%) even when
performing extreme extents of quantization with similar performance trade-offs
as with the original EXACT. Further, we present a correction to the assumptions
on the distribution of intermediate activation maps in EXACT (assumed to be
uniform) and show improved variance estimations of the quantization and
dequantization steps.

通过块状量化中间激活图，有效训练大规模图神经网络并显著减少内存消耗，同时改进和改正了 EXACT 策略的假设和方差估计。

使用改进的方差最小化的块化量化对图神经网络进行激活压缩

Activation Compression of Graph Neural Networks using Block-wise  Quantization with Improved Variance Minimization

With neural networks growing deeper and feature maps growing larger, limited
communication bandwidth with external memory (or DRAM) and power constraints
become a bottleneck in implementing network inference on mobile and edge
devices. In this paper, we propose an end-to-end differentiable bandwidth
efficient neural inference method with the activation compressed by neural data
compression method. Specifically, we propose a transform-quantization-entropy
coding pipeline for activation compression with symmetric exponential Golomb
coding and a data-dependent Gaussian entropy model for arithmetic coding.
Optimized with existing model quantization methods, low-level task of image
compression can achieve up to 19x bandwidth reduction with 6.21x energy saving.

我们提出了一种端到端可微分的带宽高效神经推理方法，通过神经数据压缩方法对激活进行压缩，可实现高达 19 倍的带宽减少和 6.21 倍的能量节省。