Visual dubbing is the process of generating lip motions of an actor in a
video to synchronise with given audio. Recent advances have made progress
towards this goal but have not been able to produce an approach suitable for
mass adoption. Existing methods are split into either person-generic or
person-specific models. Person-specific models produce results almost
indistinguishable from reality but rely on long training times using large
single-person datasets. Person-generic works have allowed for the visual
dubbing of any video to any audio without further training, but these fail to
capture the person-specific nuances and often suffer from visual artefacts. Our
method, based on data-efficient neural rendering priors, overcomes the
limitations of existing approaches. Our pipeline consists of learning a
deferred neural rendering prior network and actor-specific adaptation using
neural textures. This method allows for $\textbf{high-quality visual dubbing
with just a few seconds of data}$, that enables video dubbing for any actor -
from A-list celebrities to background actors. We show that we achieve
state-of-the-art in terms of $\textbf{visual quality}$ and
$\textbf{recognisability}$ both quantitatively, and qualitatively through two
user studies. Our prior learning and adaptation method $\textbf{generalises to
limited data}$ better and is more $\textbf{scalable}$ than existing
person-specific models. Our experiments on real-world, limited data scenarios
find that our model is preferred over all others. The project page may be found
at this https URL

利用数据效率的神经渲染先验和神经纹理进行演员特定的自适应，以达到高质量的视觉配音，只需几秒钟的数据即可实现，同时具有限数据的普遍性和可扩展性。

适用于所有人的配音：使用神经渲染先验进行高效配音

Dubbing for Everyone: Data-Efficient Visual Dubbing using Neural  Rendering Priors

Generating high-quality and person-generic visual dubbing remains a
challenge. Recent innovation has seen the advent of a two-stage paradigm,
decoupling the rendering and lip synchronization process facilitated by
intermediate representation as a conduit. Still, previous methodologies rely on
rough landmarks or are confined to a single speaker, thus limiting their
performance. In this paper, we propose DiffDub: Diffusion-based dubbing. We
first craft the Diffusion auto-encoder by an inpainting renderer incorporating
a mask to delineate editable zones and unaltered regions. This allows for
seamless filling of the lower-face region while preserving the remaining parts.
Throughout our experiments, we encountered several challenges. Primarily, the
semantic encoder lacks robustness, constricting its ability to capture
high-level features. Besides, the modeling ignored facial positioning, causing
mouth or nose jitters across frames. To tackle these issues, we employ
versatile strategies, including data augmentation and supplementary eye
guidance. Moreover, we encapsulated a conformer-based reference encoder and
motion generator fortified by a cross-attention mechanism. This enables our
model to learn person-specific textures with varying references and reduces
reliance on paired audio-visual data. Our rigorous experiments comprehensively
highlight that our ground-breaking approach outpaces existing methods with
considerable margins and delivers seamless, intelligible videos in
person-generic and multilingual scenarios.

通过 Diffusion-based dubbing 方法，我们提出了 DiffDub 自动编码器，可以实现高质量的视觉配音，包括无痕填充和人物特定纹理等创新性策略，通过全面的实验证明，我们的方法在人员通用和多语言场景中优于现有方法，并提供无缝连贯、可理解的视频。

DiffDub：使用扩充自动编码器的修复渲染器进行人物通用视觉配音

DiffDub: Person-generic Visual Dubbing Using Inpainting Renderer with  Diffusion Auto-encoder

Dubbing is a technique for translating video content from one language to
another. However, state-of-the-art visual dubbing techniques directly copy
facial expressions from source to target actors without considering
identity-specific idiosyncrasies such as a unique type of smile. We present a
style-preserving visual dubbing approach from single video inputs, which
maintains the signature style of target actors when modifying facial
expressions, including mouth motions, to match foreign languages. At the heart
of our approach is the concept of motion style, in particular for facial
expressions, i.e., the person-specific expression change that is yet another
essential factor beyond visual accuracy in face editing applications. Our
method is based on a recurrent generative adversarial network that captures the
spatiotemporal co-activation of facial expressions, and enables generating and
modifying the facial expressions of the target actor while preserving their
style. We train our model with unsynchronized source and target videos in an
unsupervised manner using cycle-consistency and mouth expression losses, and
synthesize photorealistic video frames using a layered neural face renderer.
Our approach generates temporally coherent results, and handles dynamic
backgrounds. Our results show that our dubbing approach maintains the
idiosyncratic style of the target actor better than previous approaches, even
for widely differing source and target actors.

提出了一种基于运动风格和生成对抗网络的可视化配音方法，该方法将目标演员的签名风格保留在修改面部表情（包括口型运动）以匹配外语时，并且实验结果表明，与先前的方法相比，我们的可视化配音方法更好地保持了目标演员的固有风格。