Diffusion models have demonstrated impressive performance in generating high-quality videos from text prompts or images. However, precise control over the video generation process, such as camera manipulation or content editing, remains a significant challenge. Existing methods for controlled video generation are typically limited to a single control type, lacking the flexibility to handle diverse control demands. In this paper, we introduce Diffusion as Shader (DaS), a novel approach that supports multiple video control tasks within a unified architecture. Our key insight is that achieving versatile video control necessitates leveraging 3D control signals, as videos are fundamentally 2D renderings of dynamic 3D content. Unlike prior methods limited to 2D control signals, DaS leverages 3D tracking videos as control inputs, making the video diffusion process inherently 3D-aware. This innovation allows DaS to achieve a wide range of video controls by simply manipulating the 3D tracking videos. A further advantage of using 3D tracking videos is their ability to effectively link frames, significantly enhancing the temporal consistency of the generated videos. With just 3 days of fine-tuning on 8 H800 GPUs using less than 10k videos, DaS demonstrates strong control capabilities across diverse tasks, including mesh-to-video generation, camera control, motion transfer, and object manipulation.

本文解决了视频生成过程中对生成控制的精确需求，尤其是在摄像机操作和内容编辑方面的挑战。提出的“扩散作为着色器”（DaS）方法通过使用3D跟踪视频作为控制输入，实现了多个视频控制任务的统一架构，从而提升了视频生成的灵活性和时间一致性。研究表明，DaS在不同任务（如网格到视频生成、摄像机控制、运动转移和物体操控）中展现出卓越的控制能力。

扩散作为着色器：具备3D感知的视频扩散以实现多样化视频生成控制