The fast growing capabilities of large-scale deep learning models, such as
Bert, GPT and ViT, are revolutionizing the landscape of NLP, CV and many other
domains. Training such models, however, poses an unprecedented demand for
computing power, which incurs exponentially increasing energy cost and carbon
dioxide emissions. It is thus critical to develop efficient training solutions
to reduce the training costs. Motivated by a set of key observations of inter-
and intra-layer similarities among feature maps and attentions that can be
identified from typical training processes, we propose a multi-level framework
for training acceleration. Specifically, the framework is based on three basic
operators, Coalescing, De-coalescing and Interpolation, which can be
orchestrated to build a multi-level training framework. The framework consists
of a V-cycle training process, which progressively down- and up-scales the
model size and projects the parameters between adjacent levels of models via
coalescing and de-coalescing. The key idea is that a smaller model that can be
trained for fast convergence and the trained parameters provides high-qualities
intermediate solutions for the next level larger network. The interpolation
operator is designed to break the symmetry of neurons incurred by de-coalescing
for better convergence performance. Our experiments on transformer-based
language models (e.g. Bert, GPT) as well as a vision model (e.g. DeiT) prove
that the proposed framework reduces the computational cost by about 20% on
training BERT/GPT-Base models and up to 51.6% on training the BERT-Large model
while preserving the performance.

我们提出了一个多层框架，通过合并和解散以及插值操作来加速训练，从而降低了 BERT/GPT-Base 模型的计算成本约 20%，以及 BERT-Large 模型的计算成本高达 51.6%，同时保持了性能。

加速训练 Transformer 模型的多层框架

A Multi-Level Framework for Accelerating Training Transformer Models

Out-of-distribution (OOD) detection is essential to prevent anomalous inputs
from causing a model to fail during deployment. While improved OOD detection
methods have emerged, they often rely on the final layer outputs and require a
full feedforward pass for any given input. In this paper, we propose a novel
framework, multi-level out-of-distribution detection MOOD, which exploits
intermediate classifier outputs for dynamic and efficient OOD inference. We
explore and establish a direct relationship between the OOD data complexity and
optimal exit level, and show that easy OOD examples can be effectively detected
early without propagating to deeper layers. At each exit, the OOD examples can
be distinguished through our proposed adjusted energy score, which is both
empirically and theoretically suitable for networks with multiple classifiers.
We extensively evaluate MOOD across 10 OOD datasets spanning a wide range of
complexities. Experiments demonstrate that MOOD achieves up to 71.05%
computational reduction in inference, while maintaining competitive OOD
detection performance.

本论文提出了一种新的多层级的离群检测框架 MOOD，通过中间分类器的输出进行动态且高效的离群推理，调整能量分数可以有效地检测简单的离群值，从而实现高达 71.05％的计算减少并保持竞争力的离群检测性能。