We propose GaussianTalker, a novel framework for real-time generation of
pose-controllable talking heads. It leverages the fast rendering capabilities
of 3D Gaussian Splatting (3DGS) while addressing the challenges of directly
controlling 3DGS with speech audio. GaussianTalker constructs a canonical 3DGS
representation of the head and deforms it in sync with the audio. A key insight
is to encode the 3D Gaussian attributes into a shared implicit feature
representation, where it is merged with audio features to manipulate each
Gaussian attribute. This design exploits the spatial-aware features and
enforces interactions between neighboring points. The feature embeddings are
then fed to a spatial-audio attention module, which predicts frame-wise offsets
for the attributes of each Gaussian. It is more stable than previous
concatenation or multiplication approaches for manipulating the numerous
Gaussians and their intricate parameters. Experimental results showcase
GaussianTalker's superiority in facial fidelity, lip synchronization accuracy,
and rendering speed compared to previous methods. Specifically, GaussianTalker
achieves a remarkable rendering speed of 120 FPS, surpassing previous
benchmarks. Our code is made available at
this https URL .

提出了 GaussianTalker，这是一个实时生成可调控姿势的会话头部的新框架，它利用 3D 高斯喷射（3DGS）的快速渲染能力，同时解决了直接控制 3DGS 与语音音频的挑战。

高保真实时语音合成与音频驱动下的 3D 高斯状喷溅

GaussianTalker: Real-Time High-Fidelity Talking Head Synthesis with  Audio-Driven 3D Gaussian Splatting

We present a novel character control framework that effectively utilizes
motion diffusion probabilistic models to generate high-quality and diverse
character animations, responding in real-time to a variety of dynamic
user-supplied control signals. At the heart of our method lies a
transformer-based Conditional Autoregressive Motion Diffusion Model (CAMDM),
which takes as input the character's historical motion and can generate a range
of diverse potential future motions conditioned on high-level, coarse user
control. To meet the demands for diversity, controllability, and computational
efficiency required by a real-time controller, we incorporate several key
algorithmic designs. These include separate condition tokenization,
classifier-free guidance on past motion, and heuristic future trajectory
extension, all designed to address the challenges associated with taming motion
diffusion probabilistic models for character control. As a result, our work
represents the first model that enables real-time generation of high-quality,
diverse character animations based on user interactive control, supporting
animating the character in multiple styles with a single unified model. We
evaluate our method on a diverse set of locomotion skills, demonstrating the
merits of our method over existing character controllers. Project page and
source codes: this https URL

我们提供了一个新颖的字符控制框架，利用运动扩散概率模型生成高质量且多样化的字符动画，能够实时响应各种动态用户控制信号。

驾驭扩散概率模型用于角色控制

Taming Diffusion Probabilistic Models for Character Control

Generative adversarial networks (GANs) have achieved remarkable progress in
recent years, but the continuously growing scale of models makes them
challenging to deploy widely in practical applications. In particular, for
real-time generation tasks, different devices require generators of different
sizes due to varying computing power. In this paper, we introduce slimmable
GANs (SlimGANs), which can flexibly switch the width of the generator to
accommodate various quality-efficiency trade-offs at runtime. Specifically, we
leverage multiple discriminators that share partial parameters to train the
slimmable generator. To facilitate the \textit{consistency} between generators
of different widths, we present a stepwise inplace distillation technique that
encourages narrow generators to learn from wide ones. As for class-conditional
generation, we propose a sliceable conditional batch normalization that
incorporates the label information into different widths. Our methods are
validated, both quantitatively and qualitatively, by extensive experiments and
a detailed ablation study.

本文提出了一种可以灵活调整生成器宽度以适应不同质量和效率平衡的 slimmable GANs，通过利用多个共享部分参数的判别器对生成器进行训练，并提出了一种逐步原位提纯技术以增加不同宽度之间的一致性，并通过切片条件化 BN 的方式将标签信息纳入不同的宽度中。