This paper presents a novel metric learning approach to address the
performance gap between normal and silent speech in visual speech recognition
(VSR). The difference in lip movements between the two poses a challenge for
existing VSR models, which exhibit degraded accuracy when applied to silent
speech. To solve this issue and tackle the scarcity of training data for silent
speech, we propose to leverage the shared literal content between normal and
silent speech and present a metric learning approach based on visemes.
Specifically, we aim to map the input of two speech types close to each other
in a latent space if they have similar viseme representations. By minimizing
the Kullback-Leibler divergence of the predicted viseme probability
distributions between and within the two speech types, our model effectively
learns and predicts viseme identities. Our evaluation demonstrates that our
method improves the accuracy of silent VSR, even when limited training data is
available.

本文提出一种基于 visemes 的度量学习方法，解决了视觉语音识别中正常与无声语音的表现差距问题，通过最小化预测 viseme 概率分布之间和内部的 Kullback-Leibler 散度，有效地学习和预测 viseme 身份。

基于度量学习改善正常与无声语音间的视觉语音识别差距

Improving the Gap in Visual Speech Recognition Between Normal and Silent  Speech Based on Metric Learning

In this paper, we consider the task of digitally voicing silent speech, where
silently mouthed words are converted to audible speech based on
electromyography (EMG) sensor measurements that capture muscle impulses. While
prior work has focused on training speech synthesis models from EMG collected
during vocalized speech, we are the first to train from EMG collected during
silently articulated speech. We introduce a method of training on silent EMG by
transferring audio targets from vocalized to silent signals. Our method greatly
improves intelligibility of audio generated from silent EMG compared to a
baseline that only trains with vocalized data, decreasing transcription word
error rate from 64% to 4% in one data condition and 88% to 68% in another. To
spur further development on this task, we share our new dataset of silent and
vocalized facial EMG measurements.

本文介绍了使用静态肌电信号转换为可以听到的声音的新方法，该方法与语音训练模型相比，在肌电信号在进行无声口型时的表现有了长足提升，并共享了一个新的无声和有声肌电测量数据集。