In this paper, we present an improved feedforward sequential memory networks
(FSMN) architecture, namely Deep-FSMN (DFSMN), by introducing skip connections
between memory blocks in adjacent layers. These skip connections enable the
information flow across different layers and thus alleviate the gradient
vanishing problem when building very deep structure. As a result, DFSMN
significantly benefits from these skip connections and deep structure. We have
compared the performance of DFSMN to BLSTM both with and without lower frame
rate (LFR) on several large speech recognition tasks, including English and
Mandarin. Experimental results shown that DFSMN can consistently outperform
BLSTM with dramatic gain, especially trained with LFR using CD-Phone as
modeling units. In the 2000 hours Fisher (FSH) task, the proposed DFSMN can
achieve a word error rate of 9.4% by purely using the cross-entropy criterion
and decoding with a 3-gram language model, which achieves a 1.5% absolute
improvement compared to the BLSTM. In a 20000 hours Mandarin recognition task,
the LFR trained DFSMN can achieve more than 20% relative improvement compared
to the LFR trained BLSTM. Moreover, we can easily design the lookahead filter
order of the memory blocks in DFSMN to control the latency for real-time
applications.

本文介绍了改进的前馈顺序记忆网络 DFSMN 结构，通过在相邻层的记忆块之间引入跳过连接以启用跨不同层的信息流，并消除了建立非常深层结构时的梯度消失问题。在多个大型语音识别任务中，DFSMN 的性能优于 BLSTM，尤其是使用 CD-Phone 作为建模单元和较低帧率（LFR）进行训练时，DFSMN 的改进更为显著。此外，DFSMN 的记忆块的前瞻滤波器可以轻松地设计，以控制实时应用的延迟。