The remarkable performance of Vision Transformers (ViTs) typically requires
an extremely large training cost. Existing methods have attempted to accelerate
the training of ViTs, yet typically disregard method universality with accuracy
dropping. Meanwhile, they break the training consistency of the original
transformers, including the consistency of hyper-parameters, architecture, and
strategy, which prevents them from being widely applied to different
Transformer networks. In this paper, we propose a novel token growth scheme
Token Expansion (termed ToE) to achieve consistent training acceleration for
ViTs. We introduce an "initialization-expansion-merging" pipeline to maintain
the integrity of the intermediate feature distribution of original
transformers, preventing the loss of crucial learnable information in the
training process. ToE can not only be seamlessly integrated into the training
and fine-tuning process of transformers (e.g., DeiT and LV-ViT), but also
effective for efficient training frameworks (e.g., EfficientTrain), without
twisting the original training hyper-parameters, architecture, and introducing
additional training strategies. Extensive experiments demonstrate that ToE
achieves about 1.3x faster for the training of ViTs in a lossless manner, or
even with performance gains over the full-token training baselines. Code is
available at this https URL .

我们提出了一种名为 ToE 的新型令牌增长方案，用于实现 ViTs 的一致训练加速，通过维护原始 Transformer 的中间特征分布的完整性，防止训练过程中关键可学习信息的丢失。实验结果表明，ToE 无损地实现了约 1.3 倍的 ViTs 训练加速，甚至在全令牌训练基准上取得了性能提升。