Self-supervised pre-trained models such as Wav2vec2, Hubert, and WavLM have
been shown to significantly improve many speech tasks. However, their large
memory and strong computational requirements hinder their industrial
applicability. Structured pruning is a hardware-friendly model compression
technique but usually results in a larger loss of accuracy. In this paper, we
propose a fine-grained attention head pruning method to compensate for the
performance degradation. In addition, we also introduce the straight through
estimator into the L0 regularization to further accelerate the pruned model.
Experiments on the SUPERB benchmark show that our model can achieve comparable
performance to the dense model in multiple tasks and outperforms the Wav2vec
2.0 base model on average, with 72% fewer parameters and 2 times faster
inference speed.

本文提出了一种精细的注意力头修剪方法来解决自监督预训练模型中的模型压缩问题，并介绍了直通估计量到 L0 正则化中以进一步加速修剪模型，超越 Wav2vec2.0 基准模型的表现，且具有 72% 更少的参数和两倍的推理速度。

面向任务的语音表示模型的结构化剪枝

Task-Agnostic Structured Pruning of Speech Representation Models

Probing is a popular method to discern what linguistic information is
contained in the representations of pre-trained language models. However, the
mechanism of selecting the probe model has recently been subject to intense
debate, as it is not clear if the probes are merely extracting information or
modeling the linguistic property themselves. To address this challenge, this
paper introduces a novel model-free approach to probing, by formulating probing
as a prompting task. We conduct experiments on five probing tasks and show that
our approach is comparable or better at extracting information than diagnostic
probes while learning much less on its own. We further combine the probing via
prompting approach with attention head pruning to analyze where the model
stores the linguistic information in its architecture. We then examine the
usefulness of a specific linguistic property for pre-training by removing the
heads that are essential to that property and evaluating the resulting model's
performance on language modeling.

本文提出了一种无模型的探测方法 —— 提示探测法，通过在 5 个探测任务上的实验表明，这种方法在提取信息方面与诊断探针相当或更好，并且可以自我学习得更少。此外，结合关注头修剪与提示探测法，分析模型在其架构中存储语言信息的位置，并通过删除对特定语言属性至关重要的头部来评估预训练的有用性。