Recent advances in vision Transformers (ViTs) have come with a voracious
appetite for computing power, high-lighting the urgent need to develop
efficient training methods for ViTs. Progressive learning, a training scheme
where the model capacity grows progressively during training, has started
showing its ability in efficient training. In this paper, we take a practical
step towards efficient training of ViTs by customizing and automating
progressive learning. First, we develop a strong manual baseline for
progressive learning of ViTs, by introducing momentum growth (MoGrow) to bridge
the gap brought by model growth. Then, we propose automated progressive
learning (AutoProg), an efficient training scheme that aims to achieve lossless
acceleration by automatically increasing the training overload on-the-fly; this
is achieved by adaptively deciding whether, where and how much should the model
grow during progressive learning. Specifically, we first relax the optimization
of the growth schedule to sub-network architecture optimization problem, then
propose one-shot estimation of the sub-network performance via an elastic
supernet. The searching overhead is reduced to minimal by recycling the
parameters of the supernet. Extensive experiments of efficient training on
ImageNet with two representative ViT models, DeiT and VOLO, demonstrate that
AutoProg can accelerate ViTs training by up to 85.1% with no performance drop.
Code: this https URL

该论文提出一种自适应的进阶学习自动化方案 (AutoProg)，通过在训练过程中自适应决定模型的生长量、生长位置以及生长方式等策略，从而使得模型的训练能够快速加速，且不会出现性能下降。通过 DeiT 和 VOLO 等模型在 ImageNet 上的大量实验表明，AutoProg 能够将模型的训练加速高达 85.1%。