We present a neural network structure, ControlNet, to control pretrained
large diffusion models to support additional input conditions. The ControlNet
learns task-specific conditions in an end-to-end way, and the learning is
robust even when the training dataset is small (< 50k). Moreover, training a
ControlNet is as fast as fine-tuning a diffusion model, and the model can be
trained on a personal devices. Alternatively, if powerful computation clusters
are available, the model can scale to large amounts (millions to billions) of
data. We report that large diffusion models like Stable Diffusion can be
augmented with ControlNets to enable conditional inputs like edge maps,
segmentation maps, keypoints, etc. This may enrich the methods to control large
diffusion models and further facilitate related applications.

提出一种神经网络框架 ControlNet 来控制大型预训练扩散模型以支持额外的输入条件，该模型能够以端到端的方式学习任务特定条件，并且学习具有鲁棒性，即使训练数据集很小（<50k）；此外，训练 ControlNet 的速度与调整扩散模型的速度相同，模型可以在个人设备上训练。也可以在强大的计算集群上进行训练，适用于大量（数百万至数十亿）的数据；控制网络可以与大型扩散模型一起使用，以启用诸如边缘地图、分割地图、关键点等条件输入，进一步丰富了控制大型扩散模型的方法，促进了相关应用的发展。

为文本 - 图像扩散模型添加条件控制

Adding Conditional Control to Text-to-Image Diffusion Models

Advances in technology have led to the development of methods that can create
desired visual multimedia. In particular, image generation using deep learning
has been extensively studied across diverse fields. In comparison, video
generation, especially on conditional inputs, remains a challenging and less
explored area. To narrow this gap, we aim to train our model to produce a video
corresponding to a given text description. We propose a novel training
framework, Text-to-Image-to-Video Generative Adversarial Network (TiVGAN),
which evolves frame-by-frame and finally produces a full-length video. In the
first phase, we focus on creating a high-quality single video frame while
learning the relationship between the text and an image. As the steps proceed,
our model is trained gradually on more number of consecutive frames.This
step-by-step learning process helps stabilize the training and enables the
creation of high-resolution video based on conditional text descriptions.
Qualitative and quantitative experimental results on various datasets
demonstrate the effectiveness of the proposed method.

通过训练 Text-to-Image-to-Video Generative Adversarial Network 模型，我们实现了对于给定的文本描述，生成相对应的视频，进而对深度学习、视频生成等相关领域进行研究探索。