This paper introduces a novel combination of two tasks, previously treated
separately: acoustic-to-articulatory speech inversion (AAI) and
phoneme-to-articulatory (PTA) motion estimation. We refer to this joint task as
acoustic phoneme-to-articulatory speech inversion (APTAI) and explore two
different approaches, both working speaker- and text-independently during
inference. We use a multi-task learning setup, with the end-to-end goal of
taking raw speech as input and estimating the corresponding articulatory
movements, phoneme sequence, and phoneme alignment. While both proposed
approaches share these same requirements, they differ in their way of achieving
phoneme-related predictions: one is based on frame classification, the other on
a two-staged training procedure and forced alignment. We reach competitive
performance of 0.73 mean correlation for the AAI task and achieve up to
approximately 87% frame overlap compared to a state-of-the-art text-dependent
phoneme force aligner.

引入了一种新的方法，将声学到口腔运动的转换和音素到口腔运动估计两项任务结合起来，称之为声学音素到口腔运动的反演。探索了两种不同的方法，在推理过程中都采用与说话人和文本无关的方式。使用多任务学习的模式，以端到端的目标将原始语音作为输入，估计相应的口腔运动、音素序列和音素对齐。两种方法在音素相关预测方面有所不同，一个基于帧分类，另一个采用两阶段训练过程和强制对齐。在声学到口腔运动转换任务中获得了 0.73 的平均相关性，并与现有的依赖于文本的音素强制对齐器相比，实现了高达 87% 的帧重叠。

从语音中独立估计发音器官运动和音位对齐

Speaker- and Text-Independent Estimation of Articulatory Movements and  Phoneme Alignments from Speech

Building Spoken Language Understanding (SLU) robust to Automatic Speech
Recognition (ASR) errors is an essential issue for various voice-enabled
virtual assistants. Considering that most ASR errors are caused by phonetic
confusion between similar-sounding expressions, intuitively, leveraging the
phoneme sequence of speech can complement ASR hypothesis and enhance the
robustness of SLU. This paper proposes a novel model with Cross Attention for
SLU (denoted as CASLU). The cross attention block is devised to catch the
fine-grained interactions between phoneme and word embeddings in order to make
the joint representations catch the phonetic and semantic features of input
simultaneously and for overcoming the ASR errors in downstream natural language
understanding (NLU) tasks. Extensive experiments are conducted on three
datasets, showing the effectiveness and competitiveness of our approach.
Additionally, We also validate the universality of CASLU and prove its
complementarity when combining with other robust SLU techniques.

提出了一种新的基于交叉注意力的 SLU 模型 (CASLU)，通过利用语音的音素序列来对抗 ASR 误差，实现同时捕捉输入的语音和语义特征，并在下游 NLU 任务中提高鲁棒性，大量实验证明了其有效性。