Sampling-based algorithms, which eliminate ''unimportant'' computations
during forward and/or back propagation (BP), offer potential solutions to
accelerate neural network training. However, since sampling introduces
approximations to training, such algorithms may not consistently maintain
accuracy across various tasks. In this work, we introduce a variance-controlled
adaptive sampling (VCAS) method designed to accelerate BP. VCAS computes an
unbiased stochastic gradient with fine-grained layerwise importance sampling in
data dimension for activation gradient calculation and leverage score sampling
in token dimension for weight gradient calculation. To preserve accuracy, we
control the additional variance by learning the sample ratio jointly with model
parameters during training. We assessed VCAS on multiple fine-tuning and
pre-training tasks in both vision and natural language domains. On all the
tasks, VCAS can preserve the original training loss trajectory and validation
accuracy with an up to 73.87% FLOPs reduction of BP and 49.58% FLOPs reduction
of the whole training process. The implementation is available at
this https URL .

通过使用采样算法进行神经网络训练，我们引入了一种方差控制的自适应采样（VCAS）方法，该方法可以在数据和权重梯度计算过程中使用细粒度的分层重要性采样和杠杆得分采样，以加快训练过程并降低计算量。通过在多个任务中进行评估，我们发现 VCAS 可以在保持准确性的同时，减少最高达 73.87% 的 BP 计算量和 49.58% 的整个训练过程计算量。