Inspired by the humans' cognitive ability to generalise knowledge and skills,
Self-Supervised Learning (SSL) targets at discovering general representations
from large-scale data without requiring human annotations, which is an
expensive and time consuming task. Its success in the fields of computer vision
and natural language processing have prompted its recent adoption into the
field of audio and speech processing. Comprehensive reviews summarising the
knowledge in audio SSL are currently missing. To fill this gap, in the present
work, we provide an overview of the SSL methods used for audio and speech
processing applications. Herein, we also summarise the empirical works that
exploit the audio modality in multi-modal SSL frameworks, and the existing
suitable benchmarks to evaluate the power of SSL in the computer audition
domain. Finally, we discuss some open problems and point out the future
directions on the development of audio SSL.

本文综述了自监督学习在音频处理和语音处理领域中的应用，包括方法、实验和基准数据，并讨论了未来发展方向和存在的问题。

音频自监督学习：综述

Audio Self-supervised Learning: A Survey

Recently, consistency-based methods have achieved state-of-the-art results in
semi-supervised learning (SSL). These methods always involve two roles, an
explicit or implicit teacher model and a student model, and penalize
predictions under different perturbations by a consistency constraint. However,
the weights of these two roles are tightly coupled since the teacher is
essentially an exponential moving average (EMA) of the student. In this work,
we show that the coupled EMA teacher causes a performance bottleneck. To
address this problem, we introduce Dual Student, which replaces the teacher
with another student. We also define a novel concept, stable sample, following
which a stabilization constraint is designed for our structure to be trainable.
Further, we discuss two variants of our method, which produce even higher
performance. Extensive experiments show that our method improves the
classification performance significantly on several main SSL benchmarks.
Specifically, it reduces the error rate of the 13-layer CNN from 16.84% to
12.39% on CIFAR-10 with 1k labels and from 34.10% to 31.56% on CIFAR-100 with
10k labels. In addition, our method also achieves a clear improvement in domain
adaptation.

本文介绍一种基于一致性方法来改善半监督学习（SSL）效果的新模型 —— 双生学生（Dual Student），它将原模型中的教师模型替换为另一个学生模型；还引入了一个新的概念 —— 稳定样本，利用它来设计一个稳定约束以便于模型的训练；本文还讨论了新模型的两种变体，并在 SSL 基准测试中对其进行了广泛的实验，证明了本文提出的新模型可以显著提高分类性能和领域适应性。