Despite transformers' impressive accuracy, their computational cost is often
prohibitive to use with limited computational resources. Most previous
approaches to improve inference efficiency require a separate model for each
possible computational budget. In this paper, we extend PoWER-BERT (Goyal et
al., 2020) and propose Length-Adaptive Transformer that can be used for various
inference scenarios after one-shot training. We train a transformer with
LengthDrop, a structural variant of dropout, which stochastically determines a
sequence length at each layer. We then conduct a multi-objective evolutionary
search to find a length configuration that maximizes the accuracy and minimizes
the efficiency metric under any given computational budget. Additionally, we
significantly extend the applicability of PoWER-BERT beyond sequence-level
classification into token-level classification with Drop-and-Restore process
that drops word-vectors temporarily in intermediate layers and restores at the
last layer if necessary. We empirically verify the utility of the proposed
approach by demonstrating the superior accuracy-efficiency trade-off under
various setups, including span-based question answering and text
classification. Code is available at
this https URL

本论文中提出了一种名为 Length-Adaptive Transformer 的方法，通过使用结构变体 dropout（LengthDrop）在一个转换器上一次训练即可在各种推理场景下使用，并使用多目标进化搜索查找在任何给定计算预算下最大化准确性和最小化效率度量的长度配置。此外，还通过 Drop-and-Restore 过程将 PoWER-BERT 的适用范围显着扩展到单词级分类中。该方法在各种设置下都展现了卓越的准确性和效率平衡性能。