Dysarthric speech reconstruction (DSR) aims to transform dysarthric speech
into normal speech. It still suffers from low speaker similarity and poor
prosody naturalness. In this paper, we propose a multi-modal DSR model by
leveraging neural codec language modeling to improve the reconstruction
results, especially for the speaker similarity and prosody naturalness. Our
proposed model consists of: (i) a multi-modal content encoder to extract robust
phoneme embeddings from dysarthric speech with auxiliary visual inputs; (ii) a
speaker codec encoder to extract and normalize the speaker-aware codecs from
the dysarthric speech, in order to provide original timbre and normal prosody;
(iii) a codec language model based speech decoder to reconstruct the speech
based on the extracted phoneme embeddings and normalized codecs. Evaluations on
the commonly used UASpeech corpus show that our proposed model can achieve
significant improvements in terms of speaker similarity and prosody
naturalness.

通过多模型 Dysarthric speech reconstruction（DSR）模型，利用神经编解码器语言建模提高重建结果，尤其在讲话者相似性和韵律自然性方面有显著改进。