The computation necessary for training Transformer-based language models has
skyrocketed in recent years. This trend has motivated research on efficient
training algorithms designed to improve training, validation, and downstream
performance faster than standard training. In this work, we revisit three
categories of such algorithms: dynamic architectures (layer stacking, layer
dropping), batch selection (selective backprop, RHO loss), and efficient
optimizers (Lion, Sophia). When pre-training BERT and T5 with a fixed
computation budget using such methods, we find that their training, validation,
and downstream gains vanish compared to a baseline with a fully-decayed
learning rate. We define an evaluation protocol that enables computation to be
done on arbitrary machines by mapping all computation time to a reference
machine which we call reference system time. We discuss the limitations of our
proposed protocol and release our code to encourage rigorous research in
efficient training procedures: this https URL

本文探讨了三种算法类型 (动态架构、批量选择和优化器) 在训练 Transformer-based 语言模型的效率问题，并发现这些算法并不能比传统训练方法更快的提升训练、验证和下游任务的表现。

无训练无收益：重拾用于基于 Transformer 的语言模型的高效训练算法

No Train No Gain: Revisiting Efficient Training Algorithms For  Transformer-based Language Models

Polynomial networks and factorization machines are two recently-proposed
models that can efficiently use feature interactions in classification and
regression tasks. In this paper, we revisit both models from a unified
perspective. Based on this new view, we study the properties of both models and
propose new efficient training algorithms. Key to our approach is to cast
parameter learning as a low-rank symmetric tensor estimation problem, which we
solve by multi-convex optimization. We demonstrate our approach on regression
and recommender system tasks.

本文从统一的视角重新审视了多项式网络和分解机模型，提出了高效训练算法，并将参数学习作为低秩对称张量估计问题进行求解。在回归和推荐系统任务中展示了我们的方法。