We propose a novel talking head synthesis pipeline called "DiT-Head", which
is based on diffusion transformers and uses audio as a condition to drive the
denoising process of a diffusion model. Our method is scalable and can
generalise to multiple identities while producing high-quality results. We
train and evaluate our proposed approach and compare it against existing
methods of talking head synthesis. We show that our model can compete with
these methods in terms of visual quality and lip-sync accuracy. Our results
highlight the potential of our proposed approach to be used for a wide range of
applications, including virtual assistants, entertainment, and education. For a
video demonstration of the results and our user study, please refer to our
supplementary material.

我们提出了一种基于扩散变压器的新型对话头合成流程，利用音频作为条件来驱动扩散模型的去噪过程。我们的方法具有可扩展性，可以推广到多个身份，同时产生高质量的结果。通过与现有的对话头合成方法进行比较，我们对我们提出的方法进行训练和评估，并展示了我们的模型在视觉质量和嘴唇同步准确性方面可以与这些方法竞争。我们的结果突显了我们提出的方法在包括虚拟助手、娱乐和教育在内的广泛应用中的潜力。请参阅我们的补充材料以获取结果和用户研究的视频演示。

DiT-Head：使用扩散变压器进行高分辨率说话人合成

DiT-Head: High-Resolution Talking Head Synthesis using Diffusion  Transformers

In this study, we explore Transformer-based diffusion models for image and
video generation. Despite the dominance of Transformer architectures in various
fields due to their flexibility and scalability, the visual generative domain
primarily utilizes CNN-based U-Net architectures, particularly in
diffusion-based models. We introduce GenTron, a family of Generative models
employing Transformer-based diffusion, to address this gap. Our initial step
was to adapt Diffusion Transformers (DiTs) from class to text conditioning, a
process involving thorough empirical exploration of the conditioning mechanism.
We then scale GenTron from approximately 900M to over 3B parameters, observing
significant improvements in visual quality. Furthermore, we extend GenTron to
text-to-video generation, incorporating novel motion-free guidance to enhance
video quality. In human evaluations against SDXL, GenTron achieves a 51.1% win
rate in visual quality (with a 19.8% draw rate), and a 42.3% win rate in text
alignment (with a 42.9% draw rate). GenTron also excels in the T2I-CompBench,
underscoring its strengths in compositional generation. We believe this work
will provide meaningful insights and serve as a valuable reference for future
research.

该研究探索了用于图像和视频生成的基于 Transformer 的扩散模型，并介绍了 GenTron，它是一族使用基于 Transformer 的扩散的生成模型，扩展到文本到视频生成，并在人类评估中取得了显著的视觉质量改进。

GenTron：深入探究扩散变压器在图像和视频生成中的应用

GenTron: Delving Deep into Diffusion Transformers for Image and Video  Generation

We explore a new class of diffusion models based on the transformer
architecture. We train latent diffusion models of images, replacing the
commonly-used U-Net backbone with a transformer that operates on latent
patches. We analyze the scalability of our Diffusion Transformers (DiTs)
through the lens of forward pass complexity as measured by Gflops. We find that
DiTs with higher Gflops -- through increased transformer depth/width or
increased number of input tokens -- consistently have lower FID. In addition to
possessing good scalability properties, our largest DiT-XL/2 models outperform
all prior diffusion models on the class-conditional ImageNet 512x512 and
256x256 benchmarks, achieving a state-of-the-art FID of 2.27 on the latter.

本文提出了基于 Transformer 的扩散模型。将常用的 U-Net 骨干网络替换为在潜在块上操作的 Transformer，通过 Gflops 度量前向传递复杂度分析 DiT 的可伸缩性，并证明了高 Gflops  DiT-XL/2 模型在 ImageNet 512x512 和 256x256 基准测试中均优于以前的扩散模型，后者在 256x256 基准测试上实现了先进的 FID (2.27)。