We propose multirate training of neural networks: partitioning neural network
parameters into "fast" and "slow" parts which are trained on different time
scales, where slow parts are updated less frequently. By choosing appropriate
partitionings we can obtain substantial computational speed-up for transfer
learning tasks. We show for applications in vision and NLP that we can
fine-tune deep neural networks in almost half the time, without reducing the
generalization performance of the resulting models. We analyze the convergence
properties of our multirate scheme and draw a comparison with vanilla SGD. We
also discuss splitting choices for the neural network parameters which could
enhance generalization performance when neural networks are trained from
scratch. A multirate approach can be used to learn different features present
in the data and as a form of regularization. Our paper unlocks the potential of
using multirate techniques for neural network training and provides several
starting points for future work in this area.

本文提出了一种多速率神经网络训练方法，将神经网络参数划分为 “快速” 和 “慢速” 部分，在不同的时间尺度上进行训练，从而获得传输学习任务的显著计算加速。我们通过在视觉和 NLP 应用中 fine-tune 深度神经网络，证明了我们可以在几乎一半的时间内完成调整，而不影响所得模型的泛化性能。同时，我们分析了多速率方案的收敛性质，并与普通随机梯度下降算法进行比较。本文的贡献是揭示了使用多速率技术进行神经网络训练的潜力，并为未来的工作提供了几个起点。