Pre-trained Language Models (PLMs), like BERT, with self-supervision
objectives exhibit remarkable performance and generalization across various
tasks. However, they suffer in inference latency due to their large size. To
address this issue, side branches are attached at intermediate layers, enabling
early inference of samples without requiring them to pass through all layers.
However, the challenge is to decide which layer to infer and exit each sample
so that the accuracy and latency are balanced. Moreover, the distribution of
the samples to be inferred may differ from that used for training necessitating
cross-domain adaptation. We propose an online learning algorithm named
Cross-Domain Inference in Early Exit BERT (CeeBERT) that dynamically determines
early exits of samples based on the level of confidence at each exit point.
CeeBERT learns optimal thresholds from domain-specific confidence observed at
intermediate layers on the fly, eliminating the need for labeled data.
Experimental results on five distinct datasets with BERT and ALBERT models
demonstrate CeeBERT's ability to improve latency by reducing unnecessary
computations with minimal drop in performance. By adapting to the threshold
values, CeeBERT can speed up the BERT/ALBERT models by $2\times$ - $3.5\times$
with minimal drop in accuracy.

提出了一种名为 CeeBERT 的在线学习算法，该算法通过根据每个退出点处的置信水平动态确定样本的早期退出，从而消除了标记数据的需求，并且在最少降低性能的情况下改善了延迟。