Speech-driven gesture generation is an emerging field within virtual human
creation. However, a significant challenge lies in accurately determining and
processing the multitude of input features (such as acoustic, semantic,
emotional, personality, and even subtle unknown features). Traditional
approaches, reliant on various explicit feature inputs and complex multimodal
processing, constrain the expressiveness of resulting gestures and limit their
applicability. To address these challenges, we present Persona-Gestor, a novel
end-to-end generative model designed to generate highly personalized 3D
full-body gestures solely relying on raw speech audio. The model combines a
fuzzy feature extractor and a non-autoregressive Adaptive Layer Normalization
(AdaLN) transformer diffusion architecture. The fuzzy feature extractor
harnesses a fuzzy inference strategy that automatically infers implicit,
continuous fuzzy features. These fuzzy features, represented as a unified
latent feature, are fed into the AdaLN transformer. The AdaLN transformer
introduces a conditional mechanism that applies a uniform function across all
tokens, thereby effectively modeling the correlation between the fuzzy features
and the gesture sequence. This module ensures a high level of gesture-speech
synchronization while preserving naturalness. Finally, we employ the diffusion
model to train and infer various gestures. Extensive subjective and objective
evaluations on the Trinity, ZEGGS, and BEAT datasets confirm our model's
superior performance to the current state-of-the-art approaches. Persona-Gestor
improves the system's usability and generalization capabilities, setting a new
benchmark in speech-driven gesture synthesis and broadening the horizon for
virtual human technology. Supplementary videos and code can be accessed at
this https URL

提出了一个全新的端到端生成模型，名为 Persona-Gestor，以原始语音音频为唯一输入生成高度个性化的 3D 全身手势，将模糊特征提取器和非自回归自适应层标准化 (AdaLN) 转换扩散结构相结合，并利用扩散模型进行训练和推断，通过在 Trinity、ZEGGS 和 BEAT 数据集上的广泛评估证实了其优越性能，提高了系统的可用性和泛化能力，拓宽了语音驱动手势合成的发展前景。