Dysarthric speech reconstruction (DSR) systems aim to automatically convert
dysarthric speech into normal-sounding speech. The technology eases
communication with speakers affected by the neuromotor disorder and enhances
their social inclusion. NED-based (Neural Encoder-Decoder) systems have
significantly improved the intelligibility of the reconstructed speech as
compared with GAN-based (Generative Adversarial Network) approaches, but the
approach is still limited by training inefficiency caused by the cascaded
pipeline and auxiliary tasks of the content encoder, which may in turn affect
the quality of reconstruction. Inspired by self-supervised speech
representation learning and discrete speech units, we propose a Unit-DSR
system, which harnesses the powerful domain-adaptation capacity of HuBERT for
training efficiency improvement and utilizes speech units to constrain the
dysarthric content restoration in a discrete linguistic space. Compared with
NED approaches, the Unit-DSR system only consists of a speech unit normalizer
and a Unit HiFi-GAN vocoder, which is considerably simpler without cascaded
sub-modules or auxiliary tasks. Results on the UASpeech corpus indicate that
Unit-DSR outperforms competitive baselines in terms of content restoration,
reaching a 28.2% relative average word error rate reduction when compared to
original dysarthric speech, and shows robustness against speed perturbation and
noise.

Dysarthric speech reconstruction systems have improved through NED-based approaches, but a Unit-DSR system utilizing speech units achieves better content restoration and training efficiency, outperforming baselines with robustness against speed perturbation and noise.

UNIT-DSR: 使用语音单位规范化的言语重建系统

UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit  Normalization

Speech and text are two major forms of human language. The research community
has been focusing on mapping speech to text or vice versa for many years.
However, in the field of language modeling, very little effort has been made to
model them jointly. In light of this, we explore joint language modeling for
speech units and text. Specifically, we compare different speech tokenizers to
transform continuous speech signals into discrete units and use different
methods to construct mixed speech-text data. We introduce automatic metrics to
evaluate how well the joint LM mixes speech and text. We also fine-tune the LM
on downstream spoken language understanding (SLU) tasks with different
modalities (speech or text) and test its performance to assess the model's
learning of shared representations. Our results show that by mixing speech
units and text with our proposed mixing techniques, the joint LM improves over
a speech-only baseline on SLU tasks and shows zero-shot cross-modal
transferability.

探索联合的语言建模方法，比较不同语音分词方法和语音 - 文本混合数据构建方法，通过自动指标评估联合语言模型的混合效果，在不同模态下对下游口语理解任务进行微调并测试其性能，结果表明通过混合我们提出的语音单元和文本的方法，联合语言模型在口语理解任务上超过了单独语音的基准模型，并显示出跨模态的零 - shot 传递能力。

面向语音单元和文本的联合语言建模

Toward Joint Language Modeling for Speech Units and Text

The human perception system is often assumed to recruit motor knowledge when
processing auditory speech inputs. Using articulatory modeling and deep
learning, this study examines how this articulatory information can be used for
discovering speech units in a self-supervised setting. We used vector-quantized
variational autoencoders (VQ-VAE) to learn discrete representations from
articulatory and acoustic speech data. In line with the zero-resource paradigm,
an ABX test was then used to investigate how the extracted representations
encode phonetically relevant properties. Experiments were conducted on three
different corpora in English and French. We found that articulatory information
rather organises the latent representations in terms of place of articulation
whereas the speech acoustics mainly structure the latent space in terms of
manner of articulation. We show that an optimal fusion of the two modalities
can lead to a joint representation of these phonetic dimensions more accurate
than each modality considered individually. Since articulatory information is
usually not available in a practical situation, we finally investigate the
benefit it provides when inferred from the speech acoustics in a
self-supervised manner.

使用关节建模和深度学习研究了如何在自我监督情况下使用关节信息发现语音单元，并在实验中发现联合使用关节和语音学模态可以更准确地表示这些语音学维度。

利用 VQ-VAE 从口腔和声学特征进行自监督语音单元发现

Self-supervised speech unit discovery from articulatory and acoustic features using VQ-VAE

Human infants acquire their verbal lexicon from minimal prior knowledge of
language based on the statistical properties of phonological distributions and
the co-occurrence of other sensory stimuli. In this study, we propose a novel
fully unsupervised learning method discovering speech units by utilizing
phonological information as a distributional cue and object information as a
co-occurrence cue. The proposed method can not only (1) acquire words and
phonemes from speech signals using unsupervised learning, but can also (2)
utilize object information based on multiple modalities (i.e., vision, tactile,
and auditory) simultaneously. The proposed method is based on the Nonparametric
Bayesian Double Articulation Analyzer (NPB-DAA) discovering phonemes and words
from phonological features, and Multimodal Latent Dirichlet Allocation (MLDA)
categorizing multimodal information obtained from objects. In the experiment,
the proposed method showed higher word discovery performance than the baseline
methods. In particular, words that expressed the characteristics of the object
(i.e., words corresponding to nouns and adjectives) were segmented accurately.
Furthermore, we examined how learning performance is affected by differences in
the importance of linguistic information. When the weight of the word modality
was increased, the performance was further improved compared to the fixed
condition.

通过利用音韵信息和对象信息来发现言语单元的全新无监督学习方法，能够同时根据多种模态（即视觉、触觉和听觉）使用对象信息，该方法基于非参数贝叶斯双连音分析器（NPB-DAA）发现音素和单词以及多模态潜在狄利克雷分配（MLDA）分类来自对象的多模态信息，实验表明，所提出的方法比现有基准方法具有更高的词汇发现性能。