Direct speech-to-speech translation achieves high-quality results through the
introduction of discrete units obtained from self-supervised learning. This
approach circumvents delays and cascading errors associated with model
cascading. However, talking head translation, converting audio-visual speech
(i.e., talking head video) from one language into another, still confronts
several challenges compared to audio speech: (1) Existing methods invariably
rely on cascading, synthesizing via both audio and text, resulting in delays
and cascading errors. (2) Talking head translation has a limited set of
reference frames. If the generated translation exceeds the length of the
original speech, the video sequence needs to be supplemented by repeating
frames, leading to jarring video transitions. In this work, we propose a model
for talking head translation, \textbf{TransFace}, which can directly translate
audio-visual speech into audio-visual speech in other languages. It consists of
a speech-to-unit translation model to convert audio speech into discrete units
and a unit-based audio-visual speech synthesizer, Unit2Lip, to re-synthesize
synchronized audio-visual speech from discrete units in parallel. Furthermore,
we introduce a Bounded Duration Predictor, ensuring isometric talking head
translation and preventing duplicate reference frames. Experiments demonstrate
that our proposed Unit2Lip model significantly improves synchronization (1.601
and 0.982 on LSE-C for the original and generated audio speech, respectively)
and boosts inference speed by a factor of 4.35 on LRS2. Additionally, TransFace
achieves impressive BLEU scores of 61.93 and 47.55 for Es-En and Fr-En on
LRS3-T and 100% isochronous translations.

通过自监督学习获得的离散单元，直接语音翻译实现高质量结果，克服了模型级联带来的延迟和级联错误；本文提出了一种名为 TransFace 的模型，用于直接将视听语音翻译成其他语言的视听语音，通过语音到单元翻译模型和基于单元的视听语音合成器 Unit2Lip 实现音频与视频的同步语音重新合成，进一步引入了有界时长预测器以确保等时转头说话的翻译并防止重复参考帧