Obtaining large, human labelled speech datasets to train models for emotion
recognition is a notoriously challenging task, hindered by annotation cost and
label ambiguity. In this work, we consider the task of learning embeddings for
speech classification without access to any form of labelled audio. We base our
approach on a simple hypothesis: that the emotional content of speech
correlates with the facial expression of the speaker. By exploiting this
relationship, we show that annotations of expression can be transferred from
the visual domain (faces) to the speech domain (voices) through cross-modal
distillation. We make the following contributions: (i) we develop a strong
teacher network for facial emotion recognition that achieves the state of the
art on a standard benchmark; (ii) we use the teacher to train a student, tabula
rasa, to learn representations (embeddings) for speech emotion recognition
without access to labelled audio data; and (iii) we show that the speech
emotion embedding can be used for speech emotion recognition on external
benchmark datasets. Code, models and data are available.

本篇研究提出一种基于面部表情的情感识别的无监督学习方法，通过跨模态蒸馏将面部表情标注传递到语音领域，用于语音情感识别，实现了在未标注音频数据下学习语音情感表示的目标。

野外情感语音识别中的跨模态转移

Emotion Recognition in Speech using Cross-Modal Transfer in the Wild

The teacher-student (T/S) learning has been shown effective in unsupervised
domain adaptation [1]. It is a form of transfer learning, not in terms of the
transfer of recognition decisions, but the knowledge of posteriori
probabilities in the source domain as evaluated by the teacher model. It learns
to handle the speaker and environment variability inherent in and restricted to
the speech signal in the target domain without proactively addressing the
robustness to other likely conditions. Performance degradation may thus ensue.
In this work, we advance T/S learning by proposing adversarial T/S learning to
explicitly achieve condition-robust unsupervised domain adaptation. In this
method, a student acoustic model and a condition classifier are jointly
optimized to minimize the Kullback-Leibler divergence between the output
distributions of the teacher and student models, and simultaneously, to
min-maximize the condition classification loss. A condition-invariant deep
feature is learned in the adapted student model through this procedure. We
further propose multi-factorial adversarial T/S learning which suppresses
condition variabilities caused by multiple factors simultaneously. Evaluated
with the noisy CHiME-3 test set, the proposed methods achieve relative word
error rate improvements of 44.60% and 5.38%, respectively, over a clean source
model and a strong T/S learning baseline model.

本文提出采用对抗方式的教师 - 学生学习方法，以解决语音信号在目标领域里的可靠性问题，能够学习各种条件下的非监督领域适应，并抑制多种因素导致的条件变异，达到了比较好的词错误率改进效果。

对抗式师生学习在无监督域适应中的应用

Adversarial Teacher-Student Learning for Unsupervised Domain Adaptation

Achieving high accuracy with end-to-end speech recognizers requires careful
parameter initialization prior to training. Otherwise, the networks may fail to
find a good local optimum. This is particularly true for online networks, such
as unidirectional LSTMs. Currently, the best strategy to train such systems is
to bootstrap the training from a tied-triphone system. However, this is time
consuming, and more importantly, is impossible for languages without a
high-quality pronunciation lexicon. In this work, we propose an initialization
strategy that uses teacher-student learning to transfer knowledge from a large,
well-trained, offline end-to-end speech recognition model to an online
end-to-end model, eliminating the need for a lexicon or any other linguistic
resources. We also explore curriculum learning and label smoothing and show how
they can be combined with the proposed teacher-student learning for further
improvements. We evaluate our methods on a Microsoft Cortana personal assistant
task and show that the proposed method results in a 19 % relative improvement
in word error rate compared to a randomly-initialized baseline system.

提供了一种利用师生学习技术将离线的端到端语音识别模型的知识传递到在线端到端模型中的初始化策略，用以提高在线语音识别的准确性，并结合课程学习和标签平滑实现了更好的效果。在 Microsoft Cortana 的个人助手任务中，相比随机初始化基线系统，通过所提出的方法，可以使错误率相对下降了 19%。