Transformer neural networks (TNN) demonstrated state-of-art performance on
many natural language processing (NLP) tasks, replacing recurrent neural
networks (RNNs), such as LSTMs or GRUs. However, TNNs did not perform well in
speech enhancement, whose contextual nature is different than NLP tasks, like
machine translation. Self-attention is a core building block of the
Transformer, which not only enables parallelization of sequence computation,
but also provides the constant path length between symbols that is essential to
learning long-range dependencies. In this paper, we propose a Transformer with
Gaussian-weighted self-attention (T-GSA), whose attention weights are
attenuated according to the distance between target and context symbols. The
experimental results show that the proposed T-GSA has significantly improved
speech-enhancement performance, compared to the Transformer and RNNs.

本文提出了一种高斯加权自注意力 Transformer 神经网络 (T-GSA)，通过自注意力机制的加权距离缩减，实现了显著的语音增强性能改进，在长程依赖任务中实现了并行化计算。