Generating high-quality and person-generic visual dubbing remains a
challenge. Recent innovation has seen the advent of a two-stage paradigm,
decoupling the rendering and lip synchronization process facilitated by
intermediate representation as a conduit. Still, previous methodologies rely on
rough landmarks or are confined to a single speaker, thus limiting their
performance. In this paper, we propose DiffDub: Diffusion-based dubbing. We
first craft the Diffusion auto-encoder by an inpainting renderer incorporating
a mask to delineate editable zones and unaltered regions. This allows for
seamless filling of the lower-face region while preserving the remaining parts.
Throughout our experiments, we encountered several challenges. Primarily, the
semantic encoder lacks robustness, constricting its ability to capture
high-level features. Besides, the modeling ignored facial positioning, causing
mouth or nose jitters across frames. To tackle these issues, we employ
versatile strategies, including data augmentation and supplementary eye
guidance. Moreover, we encapsulated a conformer-based reference encoder and
motion generator fortified by a cross-attention mechanism. This enables our
model to learn person-specific textures with varying references and reduces
reliance on paired audio-visual data. Our rigorous experiments comprehensively
highlight that our ground-breaking approach outpaces existing methods with
considerable margins and delivers seamless, intelligible videos in
person-generic and multilingual scenarios.

通过 Diffusion-based dubbing 方法，我们提出了 DiffDub 自动编码器，可以实现高质量的视觉配音，包括无痕填充和人物特定纹理等创新性策略，通过全面的实验证明，我们的方法在人员通用和多语言场景中优于现有方法，并提供无缝连贯、可理解的视频。