Generative spoken language models produce speech in a wide range of voices,
prosody, and recording conditions, seemingly approaching the diversity of
natural speech. However, the extent to which generated speech is acoustically
diverse remains unclear due to a lack of appropriate metrics. We address this
gap by developing lightweight metrics of acoustic diversity, which we
collectively refer to as MAD Speech. We focus on measuring five facets of
acoustic diversity: voice, gender, emotion, accent, and background noise. We
construct the metrics as a composition of specialized, per-facet embedding
models and an aggregation function that measures diversity within the embedding
space. Next, we build a series of datasets with a priori known diversity
preferences for each facet. Using these datasets, we demonstrate that our
proposed metrics achieve a stronger agreement with the ground-truth diversity
than baselines. Finally, we showcase the applicability of our proposed metrics
across several real-life evaluation scenarios. MAD Speech will be made publicly
accessible.

发展了一种声学多样性的轻量级度量标准集，通过专门的嵌入模型和一个聚合函数来衡量声学多样性，在多个真实评估场景中展示了其适用性。

MAD Speech：语音声音多样性的度量

MAD Speech: Measures of Acoustic Diversity of Speech

Interpretability research has shown that self-supervised Spoken Language
Models (SLMs) encode a wide variety of features in human speech from the
acoustic, phonetic, phonological, syntactic and semantic levels, to speaker
characteristics. The bulk of prior research on representations of phonology has
focused on segmental features such as phonemes; the encoding of suprasegmental
phonology (such as tone and stress patterns) in SLMs is not yet well
understood. Tone is a suprasegmental feature that is present in more than half
of the world's languages. This paper aims to analyze the tone encoding
capabilities of SLMs, using Mandarin and Vietnamese as case studies. We show
that SLMs encode lexical tone to a significant degree even when they are
trained on data from non-tonal languages. We further find that SLMs behave
similarly to native and non-native human participants in tone and consonant
perception studies, but they do not follow the same developmental trajectory.

自监督语言模型在编码语音的各个层面上揭示了许多特征，本文通过分析普通话和越南语来研究语言模型对音调的编码能力，发现即使在使用非语调语言的数据进行训练时，语言模型仍然在很大程度上编码了词汇音调。同时，发现语言模型在音调和辅音感知研究中表现出与母语和非母语人类参与者相似的行为，但并没有遵循相同的发展轨迹。

自我监督的口语语言模型中的声调编码

Encoding of lexical tone in self-supervised models of spoken language

We present SPECTRON, a novel approach to adapting pre-trained language models
(LMs) to perform speech continuation. By leveraging pre-trained speech
encoders, our model generates both text and speech outputs with the entire
system being trained end-to-end operating directly on spectrograms. Training
the entire model in the spectrogram domain simplifies our speech continuation
system versus existing cascade methods which use discrete speech
representations. We further show our method surpasses existing spoken language
models both in semantic content and speaker preservation while also benefiting
from the knowledge transferred from pre-existing models. Audio samples can be
found in our website this https URL

SPECTRON 是一种新的方法，用于适应预训练的语言模型来执行语音延续，它利用预训练的语音编码器，在谱图上生成文本和语音输出，并通过端到端的方式进行训练。