Multi-media communications facilitate global interaction among people.
However, despite researchers exploring cross-lingual translation techniques
such as machine translation and audio speech translation to overcome language
barriers, there is still a shortage of cross-lingual studies on visual speech.
This lack of research is mainly due to the absence of datasets containing
visual speech and translated text pairs. In this paper, we present
\textbf{AVMuST-TED}, the first dataset for \textbf{A}udio-\textbf{V}isual
\textbf{Mu}ltilingual \textbf{S}peech \textbf{T}ranslation, derived from
\textbf{TED} talks. Nonetheless, visual speech is not as distinguishable as
audio speech, making it difficult to develop a mapping from source speech
phonemes to the target language text. To address this issue, we propose
MixSpeech, a cross-modality self-learning framework that utilizes audio speech
to regularize the training of visual speech tasks. To further minimize the
cross-modality gap and its impact on knowledge transfer, we suggest adopting
mixed speech, which is created by interpolating audio and visual streams, along
with a curriculum learning strategy to adjust the mixing ratio as needed.
MixSpeech enhances speech translation in noisy environments, improving BLEU
scores for four languages on AVMuST-TED by +1.4 to +4.2. Moreover, it achieves
state-of-the-art performance in lip reading on CMLR (11.1\%), LRS2 (25.5\%),
and LRS3 (28.0\%).

本研究通过提出混合语音框架来克服跨语言视觉语音翻译中缺乏的可翻译数据问题，并提出在嘈杂环境下提高语音翻译效果的方法。