Recent Diffusion Transformers (DiTs) have shown impressive capabilities in
generating high-quality single-modality content, including images, videos, and
audio. However, it is still under-explored whether the transformer-based
diffuser can efficiently denoise the Gaussian noises towards superb multimodal
content creation. To bridge this gap, we introduce AV-DiT, a novel and
efficient audio-visual diffusion transformer designed to generate high-quality,
realistic videos with both visual and audio tracks. To minimize model
complexity and computational costs, AV-DiT utilizes a shared DiT backbone
pre-trained on image-only data, with only lightweight, newly inserted adapters
being trainable. This shared backbone facilitates both audio and video
generation. Specifically, the video branch incorporates a trainable temporal
attention layer into a frozen pre-trained DiT block for temporal consistency.
Additionally, a small number of trainable parameters adapt the image-based DiT
block for audio generation. An extra shared DiT block, equipped with
lightweight parameters, facilitates feature interaction between audio and
visual modalities, ensuring alignment. Extensive experiments on the AIST++ and
Landscape datasets demonstrate that AV-DiT achieves state-of-the-art
performance in joint audio-visual generation with significantly fewer tunable
parameters. Furthermore, our results highlight that a single shared image
generative backbone with modality-specific adaptations is sufficient for
constructing a joint audio-video generator. Our source code and pre-trained
models will be released.

通过使用基于 Transformer 的 diffuser，我们介绍了 AV-DiT，一种用于生成高质量真实视频的音频 - 视觉扩散变压器，具有多模式内容创作的显著效能，并在联合音频 - 视觉生成中具有最先进的性能。