Recent studies in speech-driven 3D talking head generation have achieved
convincing results in verbal articulations. However, generating accurate
lip-syncs degrades when applied to input speech in other languages, possibly
due to the lack of datasets covering a broad spectrum of facial movements
across languages. In this work, we introduce a novel task to generate 3D
talking heads from speeches of diverse languages. We collect a new multilingual
2D video dataset comprising over 420 hours of talking videos in 20 languages.
With our proposed dataset, we present a multilingually enhanced model that
incorporates language-specific style embeddings, enabling it to capture the
unique mouth movements associated with each language. Additionally, we present
a metric for assessing lip-sync accuracy in multilingual settings. We
demonstrate that training a 3D talking head model with our proposed dataset
significantly enhances its multilingual performance. Codes and datasets are
available at this https URL

通过多语种 2D 视频数据集，引入多语种增强模型，利用语言特定的样式嵌入，提高了 3D 说话人模型的多语种性能，并提出了度量多语种环境下的唇同步准确性指标。