Pre-trained large text-to-image (T2I) models with an appropriate text prompt
has attracted growing interests in customized images generation field. However,
catastrophic forgetting issue make it hard to continually synthesize new
user-provided styles while retaining the satisfying results amongst learned
styles. In this paper, we propose MuseumMaker, a method that enables the
synthesis of images by following a set of customized styles in a never-end
manner, and gradually accumulate these creative artistic works as a Museum.
When facing with a new customization style, we develop a style distillation
loss module to transfer the style of the whole dataset into generation of
images. It can minimize the learning biases caused by content of images, and
address the catastrophic overfitting issue induced by few-shot images. To deal
with catastrophic forgetting amongst past learned styles, we devise a dual
regularization for shared-LoRA module to optimize the direction of model
update, which could regularize the diffusion model from both weight and feature
aspects, respectively. Meanwhile, a unique token embedding corresponding to
this new style is learned by a task-wise token learning module, which could
preserve historical knowledge from past styles with the limitation of LoRA
parameter quantity. As any new user-provided style come, our MuseumMaker can
capture the nuances of the new styles while maintaining the details of learned
styles. Experimental results on diverse style datasets validate the
effectiveness of our proposed MuseumMaker method, showcasing its robustness and
versatility across various scenarios.

提出了 MuseumMaker 方法，使用预训练的大型文本到图像（T2I）模型，通过遵循一系列定制风格的方式不断地合成图像，并作为一个博物馆逐渐积累这些创造性的艺术作品。MuseumMaker 能同时捕捉新风格的细微差别，并保持已学习风格的细节，验证了其在不同情境下的稳健性和多功能性。