Self-attention network (SAN) can benefit significantly from the
bi-directional representation learning through unsupervised pretraining
paradigms such as BERT and XLNet. In this paper, we present an XLNet-like
pretraining scheme "Speech-XLNet" for unsupervised acoustic model pretraining
to learn speech representations with SAN. The pretrained SAN is finetuned under
the hybrid SAN/HMM framework. We conjecture that by shuffling the speech frame
orders, the permutation in Speech-XLNet serves as a strong regularizer to
encourage the SAN to make inferences by focusing on global structures through
its attention weights. In addition, Speech-XLNet also allows the model to
explore the bi-directional contexts for effective speech representation
learning. Experiments on TIMIT and WSJ demonstrate that Speech-XLNet greatly
improves the SAN/HMM performance in terms of both convergence speed and
recognition accuracy compared to the one trained from randomly initialized
weights. Our best systems achieve a relative improvement of 11.9% and 8.3% on
the TIMIT and WSJ tasks respectively. In particular, the best system achieves a
phone error rate (PER) of 13.3% on the TIMIT test set, which to our best
knowledge, is the lowest PER obtained from a single system.

本文提出了一种无监督预训练方案 “Speech-XLNet”，利用 XLNet 进行自注意力网络（SAN）无监督预训练得到语音表示，可以在混合 SAN / HMM 框架下微调。实验表明，Speech-XLNet 相比于从随机初始化权重训练的模型，在 TIMIT 和 WSJ 任务中都能显著提高 SAN / HMM 的性能，包括收敛速度和识别准确性。