While generative models produce high-quality images of concepts learned from
a large-scale database, a user often wishes to synthesize instantiations of
their own concepts (for example, their family, pets, or items). Can we teach a
model to quickly acquire a new concept, given a few examples? Furthermore, can
we compose multiple new concepts together? We propose Custom Diffusion, an
efficient method for augmenting existing text-to-image models. We find that
only optimizing a few parameters in the text-to-image conditioning mechanism is
sufficiently powerful to represent new concepts while enabling fast tuning (~6
minutes). Additionally, we can jointly train for multiple concepts or combine
multiple fine-tuned models into one via closed-form constrained optimization.
Our fine-tuned model generates variations of multiple, new concepts and
seamlessly composes them with existing concepts in novel settings. Our method
outperforms several baselines and concurrent works, regarding both qualitative
and quantitative evaluations, while being memory and computationally efficient.

本文介绍了一种基于 Custom Diffusion 的文本到图像生成模型，只需少量优化参数即可表示新概念并实现快速调整，还可以通过约束优化共同训练多个概念或组合多个微调模型，并在新颖的环境中将多个概念无缝组合生成，此方法在记忆和计算效能方面都占有优势。