We introduce a new approach for speech pre-training named SPIRAL which works
by learning denoising representation of perturbed data in a teacher-student
framework. Specifically, given a speech utterance, we first feed the utterance
to a teacher network to obtain corresponding representation. Then the same
utterance is perturbed and fed to a student network. The student network is
trained to output representation resembling that of the teacher. At the same
time, the teacher network is updated as moving average of student's weights
over training steps. In order to prevent representation collapse, we apply an
in-utterance contrastive loss as pre-training objective and impose position
randomization on the input to the teacher. SPIRAL achieves competitive or
better results compared to state-of-the-art speech pre-training method wav2vec
2.0, with significant reduction of training cost (80% for BASE model, 65% for
LARGE model). Furthermore, we address the problem of noise-robustness that is
critical to real-world speech applications. We propose multi-condition
pre-training by perturbing the student's input with various types of additive
noise. We demonstrate that multi-condition pre-trained SPIRAL models are more
robust to noisy speech (9.0% - 13.3% relative word error rate reduction on real
noisy test data), compared to applying multi-condition training solely in the
fine-tuning stage. Source code is available at
this https URL.

本研究提出了一种名为 SPIRAL 的语音预训练方法，它采用带噪数据的教师 - 学生框架学习去噪表示，通过对输入施加随机扰动来提高噪声鲁棒性，达到了与当前最优方法 wav2vec 2.0 相当或更好的性能并大幅降低了训练成本。

SPIRAL: 自监督扰动不变表示学习语音预训练

SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Training

We present a confidence-based single-layer feed-forward learning algorithm
SPIRAL (Spike Regularized Adaptive Learning) relying on an encoding of
activation spikes. We adaptively update a weight vector relying on confidence
estimates and activation offsets relative to previous activity. We regularize
updates proportionally to item-level confidence and weight-specific support,
loosely inspired by the observation from neurophysiology that high spike rates
are sometimes accompanied by low temporal precision. Our experiments suggest
that the new learning algorithm SPIRAL is more robust and less prone to
overfitting than both the averaged perceptron and AROW.

本文提出了一种新的基于信心的单层前馈学习算法 SPIRAL（Spike Regularized Adaptive Learning），该算法采用激活脉冲的编码，通过置信度估计和与先前活动的激活偏移进行权重向量自适应更新。实验结果表明，SPIRAL 学习算法相对于平均感知机和 AROW 方法更加稳健且不易过拟合。