Transformer-based end-to-end speech recognition has achieved great success.
However, the large footprint and computational overhead make it difficult to
deploy these models in some real-world applications. Model compression
techniques can reduce the model size and speed up inference, but the compressed
model has a fixed architecture which might be suboptimal. We propose a novel
Transformer encoder with Input-Dependent Dynamic Depth (I3D) to achieve strong
performance-efficiency trade-offs. With a similar number of layers at inference
time, I3D-based models outperform the vanilla Transformer and the static pruned
model via iterative layer pruning. We also present interesting analysis on the
gate probabilities and the input-dependency, which helps us better understand
deep encoders.

该研究提出了一种新的 Transformer 编码器模型，并利用输入依赖动态深度 (I3D) 实现了性能 - 效率的良好均衡，该方法可用于压缩模型大小并通过迭代层剪枝处理以提高模型性能，同时对门控概率和输入依赖性进行了分析以更好地理解深度编码器。