Recently, end-to-end (E2E) models become a competitive alternative to the
conventional hybrid automatic speech recognition (ASR) systems. However, they
still suffer from speaker mismatch in training and testing condition. In this
paper, we use Speech-Transformer (ST) as the study platform to investigate
speaker aware training of E2E models. We propose a model called Speaker-Aware
Speech-Transformer (SAST), which is a standard ST equipped with a speaker
attention module (SAM). The SAM has a static speaker knowledge block (SKB) that
is made of i-vectors. At each time step, the encoder output attends to the
i-vectors in the block, and generates a weighted combined speaker embedding
vector, which helps the model to normalize the speaker variations. The SAST
model trained in this way becomes independent of specific training speakers and
thus generalizes better to unseen testing speakers. We investigate different
factors of SAM. Experimental results on the AISHELL-1 task show that SAST
achieves a relative 6.5% CER reduction (CERR) over the speaker-independent (SI)
baseline. Moreover, we demonstrate that SAST still works quite well even if the
i-vectors in SKB all come from a different data source other than the acoustic
training set.

本研究利用 Speech-Transformer (SST) 研究 E2E 模型的说话人感知训练，提出了一个 Speaker-Aware Speech-Transformer (SAST) 模型，在静态的说话人知识块基础上生成加权的说话人嵌入向量，成功地将训练说话人的因素规范化，相较于独立于某个特定训练说话人的 baseline 模型，SAST 取得了相对 6.5% 的 CER 减少。