The recently proposed Conformer model has become the de facto backbone model
for various downstream speech tasks based on its hybrid attention-convolution
architecture that captures both local and global features. However, through a
series of systematic studies, we find that the Conformer architecture's design
choices are not optimal. After re-examining the design choices for both the
macro and micro-architecture of Conformer, we propose Squeezeformer which
consistently outperforms the state-of-the-art ASR models under the same
training schemes. In particular, for the macro-architecture, Squeezeformer
incorporates (i) the Temporal U-Net structure which reduces the cost of the
multi-head attention modules on long sequences, and (ii) a simpler block
structure of multi-head attention or convolution modules followed up by
feed-forward module instead of the Macaron structure proposed in Conformer.
Furthermore, for the micro-architecture, Squeezeformer (i) simplifies the
activations in the convolutional block, (ii) removes redundant Layer
Normalization operations, and (iii) incorporates an efficient depthwise
down-sampling layer to efficiently sub-sample the input signal. Squeezeformer
achieves state-of-the-art results of 7.5%, 6.5%, and 6.0% word-error-rate (WER)
on LibriSpeech test-other without external language models, which are 3.1%,
1.4%, and 0.6% better than Conformer-CTC with the same number of FLOPs. Our
code is open-sourced and available online.

通过重新审视 Conformer 架构的设计选择，我们提出了 Squeezeformer 模型，展示其在同一训练方案下一致优于当前最先进的 ASR 模型，取得了 7.5％，6.5％和 6.0％的字错率（WER）结果，比具有相同数量的 FLOPs 的 Conformer-CTC 更好 3.1％，1.4％和 0.6％。