We introduce a new approach for speech pre-training named SPIRAL which works
by learning denoising representation of perturbed data in a teacher-student
framework. Specifically, given a speech utterance, we first feed the utterance
to a teacher network to obtain corresponding representation. Then the same
utterance is perturbed and fed to a student network. The student network is
trained to output representation resembling that of the teacher. At the same
time, the teacher network is updated as moving average of student's weights
over training steps. In order to prevent representation collapse, we apply an
in-utterance contrastive loss as pre-training objective and impose position
randomization on the input to the teacher. SPIRAL achieves competitive or
better results compared to state-of-the-art speech pre-training method wav2vec
2.0, with significant reduction of training cost (80% for BASE model, 65% for
LARGE model). Furthermore, we address the problem of noise-robustness that is
critical to real-world speech applications. We propose multi-condition
pre-training by perturbing the student's input with various types of additive
noise. We demonstrate that multi-condition pre-trained SPIRAL models are more
robust to noisy speech (9.0% - 13.3% relative word error rate reduction on real
noisy test data), compared to applying multi-condition training solely in the
fine-tuning stage. Source code is available at
this https URL.

本研究提出了一种名为 SPIRAL 的语音预训练方法，它采用带噪数据的教师 - 学生框架学习去噪表示，通过对输入施加随机扰动来提高噪声鲁棒性，达到了与当前最优方法 wav2vec 2.0 相当或更好的性能并大幅降低了训练成本。

SPIRAL: 自监督扰动不变表示学习语音预训练

SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Training

Speech pre-training has primarily demonstrated efficacy on classification
tasks, while its capability of generating novel speech, similar to how GPT-2
can generate coherent paragraphs, has barely been explored. Generative Spoken
Language Modeling (GSLM) \cite{Lakhotia2021} is the only prior work addressing
the generative aspects of speech pre-training, which replaces text with
discovered phone-like units for language modeling and shows the ability to
generate meaningful novel sentences. Unfortunately, despite eliminating the
need of text, the units used in GSLM discard most of the prosodic information.
Hence, GSLM fails to leverage prosody for better comprehension, and does not
generate expressive speech. In this work, we present a prosody-aware generative
spoken language model (pGSLM). It is composed of a multi-stream transformer
language model (MS-TLM) of speech, represented as discovered unit and prosodic
feature streams, and an adapted HiFi-GAN model converting MS-TLM outputs to
waveforms. We devise a series of metrics for prosody modeling and generation,
and re-use metrics from GSLM for content modeling. Experimental results show
that the pGSLM can utilize prosody to improve both prosody and content
modeling, and also generate natural, meaningful, and coherent speech given a
spoken prompt. Audio samples can be found at this https URL
Codes and models are available at
this https URL

本文提出了一种基于多流 Transformer 和 HiFi-GAN 的韵律感知生成口语语言模型 (pGSLM)，利用韵律为生成式预训练提供了更好的理解和生成能力。实验结果表明，该模型可以显著提高韵律和内容建模，并能够生成自然、有意义和连贯的语音。

基于韵律的无文本生成式语音语言建模

Text-Free Prosody-Aware Generative Spoken Language Modeling

We introduce a self-supervised speech pre-training method called TERA, which
stands for Transformer Encoder Representations from Alteration. Recent
approaches often learn by using a single auxiliary task like contrastive
prediction, autoregressive prediction, or masked reconstruction. Unlike
previous methods, we use alteration along three orthogonal axes to pre-train
Transformer Encoders on a large amount of unlabeled speech. The model learns
through the reconstruction of acoustic frames from their altered counterpart,
where we use a stochastic policy to alter along various dimensions: time,
frequency, and magnitude. TERA can be used for speech representations
extraction or fine-tuning with downstream models. We evaluate TERA on several
downstream tasks, including phoneme classification, keyword spotting, speaker
recognition, and speech recognition. We present a large-scale comparison of
various self-supervised models. TERA achieves strong performance in the
comparison by improving upon surface features and outperforming previous
models. In our experiments, we study the effect of applying different
alteration techniques, pre-training on more data, and pre-training on various
features. We analyze different model sizes and find that smaller models are
strong representation learners than larger models, while larger models are more
effective for downstream fine-tuning than smaller models. Furthermore, we show
the proposed method is transferable to downstream datasets not used in
pre-training.

介绍了一种称为 TERA 的自监督语音预训练方法，利用一种通过沿着三个正交轴对变换来预训练 Transformer Encoders 的方法，在多个下游任务中实现了出色的性能。