Self-supervised learned models have been found to be very effective for
certain speech tasks such as automatic speech recognition, speaker
identification, keyword spotting and others. While the features are undeniably
useful in speech recognition and associated tasks, their utility in speech
enhancement systems is yet to be firmly established, and perhaps not properly
understood. In this paper, we investigate the uses of SSL representations for
single-channel speech enhancement in challenging conditions and find that they
add very little value for the enhancement task. Our constraints are designed
around on-device real-time speech enhancement -- model is causal, the compute
footprint is small. Additionally, we focus on low SNR conditions where such
models struggle to provide good enhancement. In order to systematically examine
how SSL representations impact performance of such enhancement models, we
propose a variety of techniques to utilize these embeddings which include
different forms of knowledge-distillation and pre-training.

通过对自监督学习表示进行研究，我们发现它们在挑战性条件下的单通道语音增强任务中几乎没有增加价值。为了系统地评估 SSL 表示对这些增强模型性能的影响，我们提出了多种利用这些嵌入的技术，包括不同形式的知识蒸馏和预训练。

针对设备内单通道语音增强的 Wav2Vec2 嵌入层的深入研究

A Closer Look at Wav2Vec2 Embeddings for On-Device Single-Channel Speech  Enhancement

We propose SelfVC, a training strategy to iteratively improve a voice
conversion model with self-synthesized examples. Previous efforts on voice
conversion focus on explicitly disentangling speech representations to
separately encode speaker characteristics and linguistic content. However,
disentangling speech representations to capture such attributes using
task-specific loss terms can lead to information loss by discarding finer
nuances of the original signal. In this work, instead of explicitly
disentangling attributes with loss terms, we present a framework to train a
controllable voice conversion model on entangled speech representations derived
from self-supervised learning and speaker verification models. First, we
develop techniques to derive prosodic information from the audio signal and SSL
representations to train predictive submodules in the synthesis model. Next, we
propose a training strategy to iteratively improve the synthesis model for
voice conversion, by creating a challenging training objective using
self-synthesized examples. In this training approach, the current state of the
synthesis model is used to generate voice-converted variations of an utterance,
which serve as inputs for the reconstruction task, ensuring a continuous and
purposeful refinement of the model. We demonstrate that incorporating such
self-synthesized examples during training improves the speaker similarity of
generated speech as compared to a baseline voice conversion model trained
solely on heuristically perturbed inputs. SelfVC is trained without any text
and is applicable to a range of tasks such as zero-shot voice conversion,
cross-lingual voice conversion, and controllable speech synthesis with pitch
and pace modifications. SelfVC achieves state-of-the-art results in zero-shot
voice conversion on metrics evaluating naturalness, speaker similarity, and
intelligibility of synthesized audio.

我们提出了 SelfVC，一种训练策略，通过自我合成的示例来迭代改进语音转换模型。该模型在训练过程中使用自我监督学习和说话人验证模型生成的交织的语音表示来训练具有可控性的语音转换模型，并通过创建具有挑战性的自我合成示例来不断改进模型。SelfVC 在没有文本的情况下进行训练，并适用于零样本语音转换、跨语言语音转换以及具有音高和语速修改的可控语音合成任务。在自然度、说话人相似度和可理解性的评估指标上，SelfVC 在零样本语音转换方面取得了最先进的结果。