Self-Supervised Learning (SSL) has proven to be useful in various speech
tasks. However, these methods are generally very demanding in terms of data,
memory, and computational resources. BERT-based Speech pre-Training with
Random-projection Quantizer (BEST-RQ), is an SSL method that has shown great
performance on Automatic Speech Recognition (ASR) while being simpler than
other SSL methods, such as wav2vec 2.0. Despite BEST-RQ's great performance,
details are lacking in the original paper, such as the amount of GPU/TPU hours
used in pre-training, and there is no official easy-to-use open-source
implementation. Furthermore, BEST-RQ has not been evaluated on other downstream
tasks aside from ASR and speech translation. In this work, we describe a
re-implementation of a Random-projection quantizer and perform a preliminary
study with a comparison to wav2vec 2.0 on four downstream tasks. We discuss the
details and differences of our implementation. We show that a random projection
quantizer can achieve similar downstream performance as wav2vec 2.0 while
decreasing training time by over a factor of two.

自学习（SSL）在多种语音任务中被证明是有用的。然而，这些方法通常在数据、内存和计算资源方面要求非常高。BERT 基于随机投影量化器的语音预训练 (BEST-RQ) 是一种 SSL 方法，在自动语音识别（ASR）上表现出色，同时比其他 SSL 方法（如 wav2vec 2.0）更简单。尽管 BEST-RQ 表现出色，但原文缺乏细节，比如预训练中使用的 GPU/TPU 时长以及没有官方易于使用的开源实现。此外，除了 ASR 和语音翻译之外，BEST-RQ 还没有在其他下游任务上进行评估。在这项工作中，我们描述了随机投影量化器的重新实现，并在四个下游任务上与 wav2vec 2.0 进行了初步研究比较。我们讨论了我们实现的细节和差异。我们证明了随机投影量化器可以实现与 wav2vec 2.0 相似的下游性能，同时训练时间减少了两倍以上。