Continual learning is a desirable feature in many modern machine learning
applications, which allows in-field adaptation and updating, ranging from
accommodating distribution shift, to fine-tuning, and to learning new tasks.
For applications with privacy and low latency requirements, the compute and
memory demands imposed by continual learning can be cost-prohibitive for
resource-constraint edge platforms. Reducing computational precision through
fully quantized training (FQT) simultaneously reduces memory footprint and
increases compute efficiency for both training and inference. However,
aggressive quantization especially integer FQT typically degrades model
accuracy to unacceptable levels. In this paper, we propose a technique that
leverages inexpensive Hadamard transforms to enable low-precision training with
only integer matrix multiplications. We further determine which tensors need
stochastic rounding and propose tiled matrix multiplication to enable low-bit
width accumulators. We demonstrate the effectiveness of our technique on
several human activity recognition datasets and CIFAR100 in a class incremental
learning setting. We achieve less than 0.5% and 3% accuracy degradation while
we quantize all matrix multiplications inputs down to 4-bits with 8-bit
accumulators.

通过使用廉价的哈达玛变换，我们提出了一种在只进行整数矩阵乘法的低精度训练中降低计算精度的技术，进一步确定了哪些张量需要进行随机舍入，并提出了平铺矩阵乘法来实现低位宽累加器。在几个人体活动识别数据集和 CIFAR100 的类增量学习环境中，我们展示了我们技术的有效性，当我们将所有矩阵乘法输入量化为 4 位并使用 8 位累加器时，我们实现了不到 0.5% 和 3% 的准确度降低。