Self-attention networks (SAN) have been introduced into automatic speech
recognition (ASR) and achieved state-of-the-art performance owing to its
superior ability in capturing long term dependency. One of the key ingredients
is the self-attention mechanism which can be effectively performed on the whole
utterance level. In this paper, we try to investigate whether even more
information beyond the whole utterance level can be exploited and beneficial.
We propose to apply self-attention layer with augmented memory to ASR.
Specifically, we first propose a variant model architecture which combines deep
feed-forward sequential memory network (DFSMN) with self-attention layers to
form a better baseline model compared with a purely self-attention network.
Then, we propose and compare two kinds of additional memory structures added
into self-attention layers. Experiments on large-scale LVCSR tasks show that on
four individual test sets, the DFSMN-SAN architecture outperforms vanilla SAN
encoder by 5% relatively in character error rate (CER). More importantly, the
additional memory structure provides further 5% to 11% relative improvement in
CER.

本文研究将自我关注网络和带增强记忆的自我关注层引入自动语音识别中，提出了一种 DFSMN-SAN 架构来比较两种类型的自我关注层的效果，实验证明该模型在大规模语音识别任务中相较于基准模型有较大幅度的性能提升。