We provide a two-way integration for the widely adopted ControlNet by
integrating external condition generation algorithms into a single dense
prediction method and incorporating its individually trained image generation
processes into a single model. Despite its tremendous success, the ControlNet
of a two-stage pipeline bears limitations in being not self-contained (e.g.
calls the external condition generation algorithms) with a large model
redundancy (separately trained models for different types of conditioning
inputs). Our proposed OmniControlNet consolidates 1) the condition generation
(e.g., HED edges, depth maps, user scribble, and animal pose) by a single
multi-tasking dense prediction algorithm under the task embedding guidance and
2) the image generation process for different conditioning types under the
textual embedding guidance. OmniControlNet achieves significantly reduced model
complexity and redundancy while capable of producing images of comparable
quality for conditioned text-to-image generation.

我们提供了一种双向集成的方法，将外部条件生成算法整合到一个密集预测方法中，并将其单独训练的图像生成过程合并到一个模型中。我们提出的 OmniControlNet 通过任务嵌入引导和文本嵌入引导，将条件生成和图像生成过程整合到一个模型中，从而显著降低了模型复杂性和冗余，并能够生成与条件文本到图像生成相媲美的图像。