Dysarthric speech reconstruction (DSR) systems aim to automatically convert
dysarthric speech into normal-sounding speech. The technology eases
communication with speakers affected by the neuromotor disorder and enhances
their social inclusion. NED-based (Neural Encoder-Decoder) systems have
significantly improved the intelligibility of the reconstructed speech as
compared with GAN-based (Generative Adversarial Network) approaches, but the
approach is still limited by training inefficiency caused by the cascaded
pipeline and auxiliary tasks of the content encoder, which may in turn affect
the quality of reconstruction. Inspired by self-supervised speech
representation learning and discrete speech units, we propose a Unit-DSR
system, which harnesses the powerful domain-adaptation capacity of HuBERT for
training efficiency improvement and utilizes speech units to constrain the
dysarthric content restoration in a discrete linguistic space. Compared with
NED approaches, the Unit-DSR system only consists of a speech unit normalizer
and a Unit HiFi-GAN vocoder, which is considerably simpler without cascaded
sub-modules or auxiliary tasks. Results on the UASpeech corpus indicate that
Unit-DSR outperforms competitive baselines in terms of content restoration,
reaching a 28.2% relative average word error rate reduction when compared to
original dysarthric speech, and shows robustness against speed perturbation and
noise.

Dysarthric speech reconstruction systems have improved through NED-based approaches, but a Unit-DSR system utilizing speech units achieves better content restoration and training efficiency, outperforming baselines with robustness against speed perturbation and noise.

UNIT-DSR: 使用语音单位规范化的言语重建系统

UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit  Normalization

Dysarthric speech reconstruction (DSR), which aims to improve the quality of
dysarthric speech, remains a challenge, not only because we need to restore the
speech to be normal, but also must preserve the speaker's identity. The speaker
representation extracted by the speaker encoder (SE) optimized for speaker
verification has been explored to control the speaker identity. However, the SE
may not be able to fully capture the characteristics of dysarthric speakers
that are previously unseen. To address this research problem, we propose a
novel multi-task learning strategy, i.e., adversarial speaker adaptation (ASA).
The primary task of ASA fine-tunes the SE with the speech of the target
dysarthric speaker to effectively capture identity-related information, and the
secondary task applies adversarial training to avoid the incorporation of
abnormal speaking patterns into the reconstructed speech, by regularizing the
distribution of reconstructed speech to be close to that of reference speech
with high quality. Experiments show that the proposed approach can achieve
enhanced speaker similarity and comparable speech naturalness with a strong
baseline approach. Compared with dysarthric speech, the reconstructed speech
achieves 22.3% and 31.5% absolute word error rate reduction for speakers with
moderate and moderate-severe dysarthria respectively. Our demo page is released
here: this https URL

提出了一种基于 adversarial speaker adaptation 的多任务学习策略，主要任务是 fine-tune 演讲者编码器以有效捕捉身份相关的信息，并通过应用对抗性训练来规范重建语音的分布，以避免引入异常发言模式。结果表明，该方法可在保持语音自然度的同时实现增强的演讲者相似性。