We propose a novel adversarial multi-task learning scheme, aiming at actively
curtailing the inter-talker feature variability while maximizing its senone
discriminability so as to enhance the performance of a deep neural network
(DNN) based ASR system. We call the scheme speaker-invariant training (SIT). In
SIT, a DNN acoustic model and a speaker classifier network are jointly
optimized to minimize the senone (tied triphone state) classification loss, and
simultaneously mini-maximize the speaker classification loss. A
speaker-invariant and senone-discriminative deep feature is learned through
this adversarial multi-task learning. With SIT, a canonical DNN acoustic model
with significantly reduced variance in its output probabilities is learned with
no explicit speaker-independent (SI) transformations or speaker-specific
representations used in training or testing. Evaluated on the CHiME-3 dataset,
the SIT achieves 4.99% relative word error rate (WER) improvement over the
conventional SI acoustic model. With additional unsupervised speaker
adaptation, the speaker-adapted (SA) SIT model achieves 4.86% relative WER gain
over the SA SI acoustic model.

本文提出了一种新的对抗多任务学习模型（即 Speaker-invariant training），旨在同时减少发言人的特征变异性并最大化其 Senone 区分性，以增强基于深度神经网络的自动语音识别系统性能，并通过对抗多任务学习来学习说话者不变且区分性 Senone 的深层特征。在此模型的基础上得到的结果比基于传统的对抗模型和 SI 模型组合的 ASR 系统结果更佳。