Acoustic-to-articulatory inversion (AAI) is to convert audio into articulator
movements, such as ultrasound tongue imaging (UTI) data. An issue of existing
AAI methods is only using the personalized acoustic information to derive the
general patterns of tongue motions, and thus the quality of generated UTI data
is limited. To address this issue, this paper proposes an audio-textual
diffusion model for the UTI data generation task. In this model, the inherent
acoustic characteristics of individuals related to the tongue motion details
are encoded by using wav2vec 2.0, while the ASR transcriptions related to the
universality of tongue motions are encoded by using BERT. UTI data are then
generated by using a diffusion module. Experimental results showed that the
proposed diffusion model could generate high-quality UTI data with clear tongue
contour that is crucial for the linguistic analysis and clinical assessment.
The project can be found on the
website\footnote{this https URL

本文提出了一种音频文本扩散模型用于 UTI 数据生成任务，通过编码个人的固有声学特征和与舌头运动相关的 ASR 转录，最终生成了具有清晰舌轮廓的高质量 UTI 数据，对于语言分析和临床评估至关重要。