Large diffusion-based Text-to-Image (T2I) models have shown impressive
generative powers for text-to-image generation as well as spatially conditioned
image generation. For most applications, we can train the model end-toend with
paired data to obtain photorealistic generation quality. However, to add an
additional task, one often needs to retrain the model from scratch using paired
data across all modalities to retain good generation performance. In this
paper, we tackle this issue and propose a novel strategy to scale a generative
model across new tasks with minimal compute. During our experiments, we
discovered that the variance maps of intermediate feature maps of diffusion
models capture the intensity of conditioning. Utilizing this prior information,
we propose MaxFusion, an efficient strategy to scale up text-to-image
generation models to accommodate new modality conditions. Specifically, we
combine aligned features of multiple models, hence bringing a compositional
effect. Our fusion strategy can be integrated into off-the-shelf models to
enhance their generative prowess.

本研究提出了一种名为 MaxFusion 的新策略，通过合并多个模型的对齐特征，为扩展到新模态条件的基于文本到图像生成模型提供了一个高效的伸缩方法。