Representations from pre-trained speech foundation models (SFMs) have shown
impressive performance in many downstream tasks. However, the potential
benefits of incorporating pre-trained SFM representations into speaker voice
similarity assessment have not been thoroughly investigated. In this paper, we
propose SVSNet+, a model that integrates pre-trained SFM representations to
improve performance in assessing speaker voice similarity. Experimental results
on the Voice Conversion Challenge 2018 and 2020 datasets show that SVSNet+
incorporating WavLM representations shows significant improvements compared to
baseline models. In addition, while fine-tuning WavLM with a small dataset of
the downstream task does not improve performance, using the same dataset to
learn a weighted-sum representation of WavLM can substantially improve
performance. Furthermore, when WavLM is replaced by other SFMs, SVSNet+ still
outperforms the baseline models and exhibits strong generalization ability.

本研究提出了 SVSNet + 模型，该模型整合了预训练的 SFMs 模型来改善对说话者声音相似性的评估性能，实验证明使用 WavLM 表征的 SVSNet + 相比基线模型有显著改进，将 WavLM 替换为其他 SFMs 时，SVSNet + 仍优于基线模型且表现出强大的泛化能力。

SVSNet +：使用语音基础模型的表示增强说话人声音相似性评估模型

SVSNet+: Enhancing Speaker Voice Similarity Assessment Models with  Representations from Speech Foundation Models

We propose a new model architecture specifically suited for text-to-speech
(TTS) models. We combine WavLM, a pre-trained self-supervised learning (SSL)
speech model, and the BEST-RQ vector quantization framework. We assess the
extent to which the more task-agnostic WavLM, coupled with the superior
suitability of the simplistic BEST-RQ framework for a wider array of downstream
tasks, yields favorable outcomes. Experiments on the LibriSpeech dataset with
SUPERB benchmarking assert that the proposed model significantly underperforms.
We speculate the underlying reason for this performance is related to the
difference between featurizing raw audio waveforms and spectrograms with a
quantizer. We discuss the limitations of this approach to better guide future
advancements in TTS.

我们提出了一种新的适用于文本到语音（TTS）模型的模型架构，将预训练的自监督学习（SSL）语音模型 WavLM 与简单的 BEST-RQ 向量量化框架相结合，通过在 LibriSpeech 数据集上的 SUPERB 基准测试实验证明，该模型表现明显不佳，我们推测这种表现偏差与使用量化器对原始音频波形和频谱图进行特征处理之间的差异有关，我们讨论了这种方法的局限性以更好地引导 TTS 的未来发展。

一个实验研究：评估 WavLM 和 BEST-RQ 的组合框架在文本到语音合成中的应用

An Experimental Study: Assessing the Combined Framework of WavLM and  BEST-RQ for Text-to-Speech Synthesis

The generation of co-speech gestures for digital humans is an emerging area
in the field of virtual human creation. Prior research has made progress by
using acoustic and semantic information as input and adopting classify method
to identify the person's ID and emotion for driving co-speech gesture
generation. However, this endeavour still faces significant challenges. These
challenges go beyond the intricate interplay between co-speech gestures, speech
acoustic, and semantics; they also encompass the complexities associated with
personality, emotion, and other obscure but important factors. This paper
introduces "diffmotion-v2," a speech-conditional diffusion-based and
non-autoregressive transformer-based generative model with WavLM pre-trained
model. It can produce individual and stylized full-body co-speech gestures only
using raw speech audio, eliminating the need for complex multimodal processing
and manually annotated. Firstly, considering that speech audio not only
contains acoustic and semantic features but also conveys personality traits,
emotions, and more subtle information related to accompanying gestures, we
pioneer the adaptation of WavLM, a large-scale pre-trained model, to extract
low-level and high-level audio information. Secondly, we introduce an adaptive
layer norm architecture in the transformer-based layer to learn the
relationship between speech information and accompanying gestures. Extensive
subjective evaluation experiments are conducted on the Trinity, ZEGGS, and BEAT
datasets to confirm the WavLM and the model's ability to synthesize natural
co-speech gestures with various styles.

该研究介绍了使用 WavLM 预训练模型的 speech-conditional diffusion-based 和 non-autoregressive transformer-based 生成模型 “diffmotion-v2”，通过原始语音音频产生个体化和风格化的全身共说手势，消除了复杂的多模态处理和手动注释的需求，实现了低级和高级音频信息的提取及语音信息与共说手势之间的关系学习。在多个数据集上进行了大量评估实验以验证 WavLM 和模型合成具有不同风格的自然共说手势的能力。