Self-supervised learning (SSL) models reshaped our approach to speech,
language and vision. However their huge size and the opaque relations between
their layers and tasks result in slow inference and network overthinking, where
predictions made from the last layer of large models is worse than those made
from intermediate layers. Early exit (EE) strategies can solve both issues by
dynamically reducing computations at inference time for certain samples.
Although popular for classification tasks in vision and language, EE has seen
less use for sequence-to-sequence speech recognition (ASR) tasks where outputs
from early layers are often degenerate. This challenge is further compounded
when speech SSL models are applied on out-of-distribution (OOD) data. This
paper first shows that SSL models do overthinking in ASR. We then motivate
further research in EE by computing an optimal bound for performance versus
speed trade-offs. To approach this bound we propose two new strategies for ASR:
(1) we adapt the recently proposed patience strategy to ASR; and (2) we design
a new EE strategy specific to ASR that performs better than all strategies
previously introduced.

本文介绍了一种应用于 ASR 任务的早期退出策略（EE），借此解决自我监督学习模型在大规模任务中存在的推理速度慢和网络反应过度的问题，并提出了两种新的策略来优化性能和速度之间的平衡。