Foundation models (FMs), that are trained on broad data at scale and are
adaptable to a wide range of downstream tasks, have brought large interest in
the research community. Benefiting from the diverse data sources such as
different modalities, languages and application domains, foundation models have
demonstrated strong generalization and knowledge transfer capabilities. In this
paper, we present a pioneering study towards building an efficient solution for
FM-based speech recognition systems. We adopt the recently developed
self-supervised BEST-RQ for pretraining, and propose the joint finetuning with
both source and unsupervised target domain data using JUST Hydra. The FM
encoder adapter and decoder are then finetuned to the target domain with a
small amount of supervised in-domain data. On a large-scale YouTube and Voice
Search task, our method is shown to be both data and model parameter efficient.
It achieves the same quality with only 21.6M supervised in-domain data and
130.8M finetuned parameters, compared to the 731.1M model trained from scratch
on additional 300M supervised in-domain data.

本论文采用自监督预训练方法 BEST-RQ，并使用 JUST Hydra 进行源域和无监督目标域数据的联合微调，将 FM 编码器适配器和解码器微调到目标域，使用少量带标记的域内数据，实现了在大规模 YouTube 和语音搜索任务中既数据又模型参数的高效性，在仅使用 2160 万个带标签的域内数据和 130.8 M 的微调参数的情况下，与另外 3 亿带标签的域内数据从头训练 731.1M 模型的效果相同。