Recently, Conformer as a backbone network for end-to-end automatic speech
recognition achieved state-of-the-art performance. The Conformer block
leverages a self-attention mechanism to capture global information, along with
a convolutional neural network to capture local information, resulting in
improved performance. However, the Conformer-based model encounters an issue
with the self-attention mechanism, as computational complexity grows
quadratically with the length of the input sequence. Inspired by previous
Connectionist Temporal Classification (CTC) guided blank skipping during
decoding, we introduce intermediate CTC outputs as guidance into the
downsampling procedure of the Conformer encoder. We define the frame with
non-blank output as key frame. Specifically, we introduce the key frame-based
self-attention (KFSA) mechanism, a novel method to reduce the computation of
the self-attention mechanism using key frames. The structure of our proposed
approach comprises two encoders. Following the initial encoder, we introduce an
intermediate CTC loss function to compute the label frame, enabling us to
extract the key frames and blank frames for KFSA. Furthermore, we introduce the
key frame-based downsampling (KFDS) mechanism to operate on high-dimensional
acoustic features directly and drop the frames corresponding to blank labels,
which results in new acoustic feature sequences as input to the second encoder.
By using the proposed method, which achieves comparable or higher performance
than vanilla Conformer and other similar work such as Efficient Conformer.
Meantime, our proposed method can discard more than 60\% useless frames during
model training and inference, which will accelerate the inference speed
significantly. This work code is available in
{this https URL}

我们提出了一种使用关键帧的自注意力机制和下采样机制的方法，能够降低 Conformer 模型中自注意力机制的计算复杂度，并取得与基准模型相当或更高的性能。同时，我们的方法在模型训练和推断过程中，可以丢弃超过 60% 的无用帧，从而显著加速推断速度。