In this paper, we propose a new Self-Supervised Learning (SSL) algorithm
called data2vec-aqc, for speech representation learning from unlabeled speech
data. Our goal is to improve SSL for speech in domains where both unlabeled and
labeled data are limited. Building on the recently introduced data2vec, we
introduce additional modules to the data2vec framework that leverage the
benefit of data augmentations, quantized representations, and clustering. The
interaction between these modules helps solve the cross-contrastive loss as an
additional self-supervised objective. data2vec-aqc achieves up to 14.1% and
20.9% relative WER improvement over the existing state-of-the-art data2vec
system on the test-clean and test-other sets, respectively, of LibriSpeech,
without the use of any language model. Our proposed model also achieves up to
17.8% relative WER improvement over the baseline data2vec when fine-tuned on
Switchboard data.

提出一种名为 data2vec-aqc 的自监督学习算法，利用数据增强，量化表示和聚类等技术来提高语音领域中有限的未标记和标记数据的质量。该算法在 LibriSpeech 的测试数据集上实现了 14.1％至 20.9％的相对词错误率改进，并且在 SwitchBoard 数据上微调时相对于基线 data2vec 获得了高达 17.8％的相对 WER 改进。

data2vec-aqc：在教师 - 学生培训环境中寻找合适的助教

data2vec-aqc: Search for the right Teaching Assistant in the Teacher-Student training setup

While Self-Supervised Learning has helped reap the benefit of the scale from
the available unlabeled data, the learning paradigms are continuously being
bettered. We present a new pre-training strategy named ccc-wav2vec 2.0, which
uses clustering and an augmentation-based cross-contrastive loss as its
self-supervised objective. Through the clustering module, we scale down the
influence of those negative examples that are highly similar to the positive.
The Cross-Contrastive loss is computed between the encoder output of the
original sample and the quantizer output of its augmentation and vice-versa,
bringing robustness to the pre-training strategy. ccc-wav2vec 2.0 achieves up
to 15.6% and 12.7% relative WER improvement over the baseline wav2vec 2.0 on
the test-clean and test-other sets, respectively, of LibriSpeech, without the
use of any language model. The proposed method also achieves up to 14.9%
relative WER improvement over the baseline wav2vec 2.0 when fine-tuned on
Switchboard data. We make all our codes publicly available on GitHub.

提出了一种名为 ccc-wav2vec 2.0 的新的自监督预训练策略，该方法使用聚类和基于数据增强的相交对比损失作为自监督目标，并取得了约 15.6% 和 12.7% 的 WER 相对改进，也可在 Switchboard 数据上获得最高 14.9% 的 WER 相对改进。