While large transformer-based models have exhibited remarkable performance in
speaker-independent speech recognition, their large size and computational
requirements make them expensive or impractical to use in resource-constrained
settings. In this work, we propose a low-rank adaptive compression technique
called AdaPTwin that jointly compresses product-dependent pairs of weight
matrices in the transformer attention layer. Our approach can prioritize the
compressed model's performance on a specific speaker while maintaining
generalizability to new speakers and acoustic conditions. Notably, our
technique requires only 8 hours of speech data for fine-tuning, which can be
accomplished in under 20 minutes, making it highly cost-effective compared to
other compression methods. We demonstrate the efficacy of our approach by
compressing the Whisper and Distil-Whisper models by up to 45% while incurring
less than a 2% increase in word error rate.

在资源受限的环境中，我们提出一种名为 AdaPTwin 的低秩自适应压缩技术，它可以压缩转换器注意层中的产品相关权重矩阵对，以降低大型转换器模型的计算和存储开销，并实现对新说话人和声学环境的泛化性能。这种压缩技术只需 8 小时的语音数据进行微调，时间不到 20 分钟，与其他压缩方法相比成本极低，并且在压缩 Whisper 和 Distil-Whisper 模型时，词错误率仅增加不到 2%。