We present a neural network structure, ControlNet, to control pretrained
large diffusion models to support additional input conditions. The ControlNet
learns task-specific conditions in an end-to-end way, and the learning is
robust even when the training dataset is small (< 50k). Moreover, training a
ControlNet is as fast as fine-tuning a diffusion model, and the model can be
trained on a personal devices. Alternatively, if powerful computation clusters
are available, the model can scale to large amounts (millions to billions) of
data. We report that large diffusion models like Stable Diffusion can be
augmented with ControlNets to enable conditional inputs like edge maps,
segmentation maps, keypoints, etc. This may enrich the methods to control large
diffusion models and further facilitate related applications.

提出一种神经网络框架 ControlNet 来控制大型预训练扩散模型以支持额外的输入条件，该模型能够以端到端的方式学习任务特定条件，并且学习具有鲁棒性，即使训练数据集很小（<50k）；此外，训练 ControlNet 的速度与调整扩散模型的速度相同，模型可以在个人设备上训练。也可以在强大的计算集群上进行训练，适用于大量（数百万至数十亿）的数据；控制网络可以与大型扩散模型一起使用，以启用诸如边缘地图、分割地图、关键点等条件输入，进一步丰富了控制大型扩散模型的方法，促进了相关应用的发展。