Speech in-painting is the task of regenerating missing audio contents using
reliable context information. Despite various recent studies in multi-modal
perception of audio in-painting, there is still a need for an effective
infusion of visual and auditory information in speech in-painting. In this
paper, we introduce a novel sequence-to-sequence model that leverages the
visual information to in-paint audio signals via an encoder-decoder
architecture. The encoder plays the role of a lip-reader for facial recordings
and the decoder takes both encoder outputs as well as the distorted audio
spectrograms to restore the original speech. Our model outperforms an
audio-only speech in-painting model and has comparable results with a recent
multi-modal speech in-painter in terms of speech quality and intelligibility
metrics for distortions of 300 ms to 1500 ms duration, which proves the
effectiveness of the introduced multi-modality in speech in-painting.

该研究提出了一种新颖的序列到序列模型，通过编码器 - 解码器架构的方式利用视觉信息来修复音频信号，证明了多模态方法在语音修复中的有效性。

序列到序列多模态语音修复

Sequence-to-Sequence Multi-Modal Speech In-Painting

The process of reconstructing missing parts of speech audio from context is
called speech in-painting. Human perception of speech is inherently
multi-modal, involving both audio and visual (AV) cues. In this paper, we
introduce and study a sequence-to-sequence (seq2seq) speech in-painting model
that incorporates AV features. Our approach extends AV speech in-painting
techniques to scenarios where both audio and visual data may be jointly
corrupted. To achieve this, we employ a multi-modal training paradigm that
boosts the robustness of our model across various conditions involving acoustic
and visual distortions. This makes our distortion-aware model a plausible
solution for real-world challenging environments. We compare our method with
existing transformer-based and recurrent neural network-based models, which
attempt to reconstruct missing speech gaps ranging from a few milliseconds to
over a second. Our experimental results demonstrate that our novel seq2seq
architecture outperforms the state-of-the-art transformer solution by 38.8% in
terms of enhancing speech quality and 7.14% in terms of improving speech
intelligibility. We exploit a multi-task learning framework that simultaneously
performs lip-reading (transcribing video components to text) while
reconstructing missing parts of the associated speech.

通过引入 AV 特征，我们研究了一种序列到序列（seq2seq）语音修复模型，将语音修复技术扩展到同时受到音频和视觉数据干扰的情境中，实验证明我们的模型在提高语音质量和可理解性方面优于现有的变压器解决方案。同时，我们利用多任务学习框架，在恢复相关语音的同时进行口型阅读（将视频组件转录为文本）。