While self-supervised speech representation learning (SSL) models serve a
variety of downstream tasks, these models have been observed to overfit to the
domain from which the unlabelled data originates. To alleviate this issue, we
propose PADA (Pruning Assisted Domain Adaptation) and zero out redundant
weights from models pre-trained on large amounts of out-of-domain (OOD) data.
Intuitively, this helps to make space for the target-domain ASR finetuning. The
redundant weights can be identified through various pruning strategies which
have been discussed in detail as a part of this work. Specifically, we
investigate the effect of the recently discovered Task-Agnostic and Task-Aware
pruning on PADA and propose a new pruning paradigm based on the latter, which
we call Cross-Domain Task-Aware Pruning (CD-TAW). CD-TAW obtains the initial
pruning mask from a well fine-tuned OOD model, which makes it starkly different
from the rest of the pruning strategies discussed in the paper. Our proposed
CD-TAW methodology achieves up to 20.6% relative WER improvement over our
baseline when fine-tuned on a 2-hour subset of Switchboard data without
language model (LM) decoding. Furthermore, we conduct a detailed analysis to
highlight the key design choices of our proposed method.

本文提出了一种名为 PADA 的方法，其中通过修剪已预训练在大量跨领域（OOD）数据上的模型中的冗余权重，帮助目标领域 ASR 微调。通过各种修剪策略的比较研究，本文探讨了基于任务对称和不对称修剪对 PADA 的影响，并提出了一种称为基于交叉领域任务感知修剪（CD-TAW）的新的修剪范式。CD-TAW 可以获得来自精细调整的 OOD 模型的初始修剪掩模。实验表明，CD-TAW 方法在没有语言模型解码的 Switchboard 数据的 2 小时子集上进行微调时相对 WER 提高了最多 20.6％。