Controllable text-to-image (T2I) diffusion models generate images conditioned
on both text prompts and semantic inputs of other modalities like edge maps.
Nevertheless, current controllable T2I methods commonly face challenges related
to efficiency and faithfulness, especially when conditioning on multiple inputs
from either the same or diverse modalities. In this paper, we propose a novel
Flexible and Efficient method, FlexEControl, for controllable T2I generation.
At the core of FlexEControl is a unique weight decomposition strategy, which
allows for streamlined integration of various input types. This approach not
only enhances the faithfulness of the generated image to the control, but also
significantly reduces the computational overhead typically associated with
multimodal conditioning. Our approach achieves a reduction of 41% in trainable
parameters and 30% in memory usage compared with Uni-ControlNet. Moreover, it
doubles data efficiency and can flexibly generate images under the guidance of
multiple input conditions of various modalities.

提出了一种新颖的灵活高效的可控文本到图像生成方法 FlexEControl，其核心是一种独特的权重分解策略，能够有效地整合不同类型的输入，提高了生成图像与控制之间的准确性，同时显著降低了与多模态条件相关的计算开销。相比 Uni-ControlNet，该方法在可训练参数和内存使用方面分别减少了 41% 和 30%，且能够灵活生成在多种输入条件下引导的图像。