Albeit great performance of Transformer-based speech selfsupervised learning
(SSL) models, their large parameter size and computational cost make them
unfavorable to utilize. In this study, we propose to compress the speech SSL
models by distilling speech temporal relation (STaR). Unlike previous works
that directly match the representation for each speech frame, STaR distillation
transfers temporal relation between speech frames, which is more suitable for
lightweight student with limited capacity. We explore three STaR distillation
objectives and select the best combination as the final STaR loss. Our model
distilled from HuBERT BASE achieves an overall score of 79.8 on SUPERB
benchmark, the best performance among models with up to 27 million parameters.
We show that our method is applicable across different speech SSL models and
maintains robust performance with further reduced parameters.

通过使用 STaR（speech temporal relation）蒸馏方法，能够在参数较少的情况下压缩和保持语音自监督学习模型的表现，该方法适用于不同的语音自监督学习模型并具有鲁棒的性能。

STaR: 为轻量级语音自监督学习模型提取语音时序相关性

STaR: Distilling Speech Temporal Relation for Lightweight Speech  Self-Supervised Learning Models

Self-supervised speech representation learning has shown promising results in
various speech processing tasks. However, the pre-trained models, e.g., HuBERT,
are storage-intensive Transformers, limiting their scope of applications under
low-resource settings. To this end, we propose LightHuBERT, a once-for-all
Transformer compression framework, to find the desired architectures
automatically by pruning structured parameters. More precisely, we create a
Transformer-based supernet that is nested with thousands of weight-sharing
subnets and design a two-stage distillation strategy to leverage the
contextualized latent representations from HuBERT. Experiments on automatic
speech recognition (ASR) and the SUPERB benchmark show the proposed LightHuBERT
enables over $10^9$ architectures concerning the embedding dimension, attention
dimension, head number, feed-forward network ratio, and network depth.
LightHuBERT outperforms the original HuBERT on ASR and five SUPERB tasks with
the HuBERT size, achieves comparable performance to the teacher model in most
tasks with a reduction of 29% parameters, and obtains a $3.5\times$ compression
ratio in three SUPERB tasks, e.g., automatic speaker verification, keyword
spotting, and intent classification, with a slight accuracy loss. The code and
pre-trained models are available at
this https URL.

我们提出了 once-for-all Transformer 压缩框架 LightHuBERT，通过剪枝结构化参数自动找到所需的结构，成功地将嵌套数千个共享权重子网的基于 Transformer 的超网进行了设计，并使用两阶段蒸馏策略从 HuBERT 利用上下文化的潜在表征。在自动语音识别（ASR）和 SUPERB 基准测试上实验表明，LightHuBERT 实现了超过 $10^9$ 种结构，深度，宽度，注意力维度，前馈网络比例和网络深度，在 ASR 和五个 SUPERB 任务上表现优于原始的 HuBERT，与该教师模型在大多数任务中表现相当，在三个 SUPERB 任务中具有 $3.5	imes$ 的压缩比，即自动说话人验证、关键词检测和意图分类，在稍微减少准确率的情况下，可以实现 29％ 的参数减少，并提供代码和预训练模型。

LightHuBERT: 一种轻量级可配置的语音表征学习方法，采用一次性隐藏单元 BERT

LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT

In this paper, we describe our submissions to the ZeroSpeech 2021 Challenge
and SUPERB benchmark. Our submissions are based on the recently proposed
FaST-VGS model, which is a Transformer-based model that learns to associate raw
speech waveforms with semantically related images, all without the use of any
transcriptions of the speech. Additionally, we introduce a novel extension of
this model, FaST-VGS+, which is learned in a multi-task fashion with a masked
language modeling objective in addition to the visual grounding objective. On
ZeroSpeech 2021, we show that our models perform competitively on the ABX task,
outperform all other concurrent submissions on the Syntactic and Semantic
tasks, and nearly match the best system on the Lexical task. On the SUPERB
benchmark, we show that our models also achieve strong performance, in some
cases even outperforming the popular wav2vec2.0 model.

本研究基于最近提出的 FaST-VGS 模型，该模型是一种基于 Transformer 的模型，学习将原始语音波形与语义相关的图像相关联，同时引入了一种新颖的扩展模型 FaST-VGS +，该模型在多任务训练中学习了掩码语言建模目标和视觉基础目标。我们的研究在 ZeroSpeech 2021 Challenge 和 SUPERB benchmark 上表现强劲，几乎在 Lexical 任务上与最佳系统相媲美。