The fast growing capabilities of large-scale deep learning models, such as
Bert, GPT and ViT, are revolutionizing the landscape of NLP, CV and many other
domains. Training such models, however, poses an unprecedented demand for
computing power, which incurs exponentially increasing energy cost and carbon
dioxide emissions. It is thus critical to develop efficient training solutions
to reduce the training costs. Motivated by a set of key observations of inter-
and intra-layer similarities among feature maps and attentions that can be
identified from typical training processes, we propose a multi-level framework
for training acceleration. Specifically, the framework is based on three basic
operators, Coalescing, De-coalescing and Interpolation, which can be
orchestrated to build a multi-level training framework. The framework consists
of a V-cycle training process, which progressively down- and up-scales the
model size and projects the parameters between adjacent levels of models via
coalescing and de-coalescing. The key idea is that a smaller model that can be
trained for fast convergence and the trained parameters provides high-qualities
intermediate solutions for the next level larger network. The interpolation
operator is designed to break the symmetry of neurons incurred by de-coalescing
for better convergence performance. Our experiments on transformer-based
language models (e.g. Bert, GPT) as well as a vision model (e.g. DeiT) prove
that the proposed framework reduces the computational cost by about 20% on
training BERT/GPT-Base models and up to 51.6% on training the BERT-Large model
while preserving the performance.

我们提出了一个多层框架，通过合并和解散以及插值操作来加速训练，从而降低了 BERT/GPT-Base 模型的计算成本约 20%，以及 BERT-Large 模型的计算成本高达 51.6%，同时保持了性能。