Albeit great performance of Transformer-based speech selfsupervised learning
(SSL) models, their large parameter size and computational cost make them
unfavorable to utilize. In this study, we propose to compress the speech SSL
models by distilling speech temporal relation (STaR). Unlike previous works
that directly match the representation for each speech frame, STaR distillation
transfers temporal relation between speech frames, which is more suitable for
lightweight student with limited capacity. We explore three STaR distillation
objectives and select the best combination as the final STaR loss. Our model
distilled from HuBERT BASE achieves an overall score of 79.8 on SUPERB
benchmark, the best performance among models with up to 27 million parameters.
We show that our method is applicable across different speech SSL models and
maintains robust performance with further reduced parameters.

通过使用 STaR（speech temporal relation）蒸馏方法，能够在参数较少的情况下压缩和保持语音自监督学习模型的表现，该方法适用于不同的语音自监督学习模型并具有鲁棒的性能。

STaR: 为轻量级语音自监督学习模型提取语音时序相关性

STaR: Distilling Speech Temporal Relation for Lightweight Speech  Self-Supervised Learning Models

Transformer-based speech self-supervised learning (SSL) models, such as
HuBERT, show surprising performance in various speech processing tasks.
However, huge number of parameters in speech SSL models necessitate the
compression to a more compact model for wider usage in academia or small
companies. In this study, we suggest to reuse attention maps across the
Transformer layers, so as to remove key and query parameters while retaining
the number of layers. Furthermore, we propose a novel masking distillation
strategy to improve the student model's speech representation quality. We
extend the distillation loss to utilize both masked and unmasked speech frames
to fully leverage the teacher model's high-quality representation. Our
universal compression strategy yields the student model that achieves phoneme
error rate (PER) of 7.72% and word error rate (WER) of 9.96% on the SUPERB
benchmark.

本研究提出基于 Transformer 的语音自监督学习模型压缩方法，包括重用注意力矩阵并采用新型蒸馏策略。我们的通用压缩策略可在 SUPERB 基准测试中实现 7.72％ 的音素错误率和 9.96％ 的单词错误率。