With the rapid increase in model size and the growing importance of various
fine-tuning applications, lightweight training has become crucial. Since the
backward pass is twice as expensive as the forward pass, optimizing
backpropagation is particularly important. However, modifications to this
process can lead to suboptimal convergence, so training optimization should
minimize perturbations, which is a highly challenging task. In this study, we
introduce a novel optimization strategy called Hadamard Low-rank Quantization
(HLQ), focusing on reducing the cost of backpropagation in convolutional and
linear layers. We first analyze the sensitivity of gradient computation with
respect to activation and weight, and judiciously design the HLQ pipeline to
apply 4-bit Hadamard quantization to the activation gradient and Hadamard
low-rank approximation to the weight gradient. This combination was found to be
the best for maximizing benefits, and our extensive experiments demonstrate the
outstanding performance of HLQ in both training from scratch and fine-tuning,
achieving significant memory savings and acceleration on real GPUs with
negligible quality degradation.

通过引入一种名为 Hadamard Low-rank Quantization (HLQ) 的优化策略，该研究主要关注在降低卷积和线性层的反向传播成本，在从头训练和微调过程中实现显著的内存节省和加速，并且获得了卓越的性能表现。