Conformer has proven to be effective in many speech processing tasks. It
combines the benefits of extracting local dependencies using convolutions and
global dependencies using self-attention. Inspired by this, we propose a more
flexible, interpretable and customizable encoder alternative, Branchformer,
with parallel branches for modeling various ranged dependencies in end-to-end
speech processing. In each encoder layer, one branch employs self-attention or
its variant to capture long-range dependencies, while the other branch utilizes
an MLP module with convolutional gating (cgMLP) to extract local relationships.
We conduct experiments on several speech recognition and spoken language
understanding benchmarks. Results show that our model outperforms both
Transformer and cgMLP. It also matches with or outperforms state-of-the-art
results achieved by Conformer. Furthermore, we show various strategies to
reduce computation thanks to the two-branch architecture, including the ability
to have variable inference complexity in a single trained model. The weights
learned for merging branches indicate how local and global dependencies are
utilized in different layers, which benefits model designing.

本文章提出一种基于自注意力机制和 MLP 模块的可定制语音转录模型 Branchformer，其可以同时提取全局和局部依赖关系，并且在多项基准测试上均胜过了 Transformer 和 cgMLP，与 Conformer 的表现相当。此外，该模型具有双分支结构，可以降低计算复杂度。