Given a small set of images of a specific subject, subject-driven
text-to-image generation aims to generate customized images of the subject
according to new text descriptions, which has attracted increasing attention in
the community recently. Current subject-driven text-to-image generation methods
are mainly based on finetuning a pretrained large-scale text-to-image
generation model. However, these finetuning methods map the images of the
subject into an embedding highly entangled with subject-identity-unrelated
information, which may result in the inconsistency between the generated images
and the text descriptions and the changes in the subject identity. To tackle
the problem, we propose DisenBooth, a disentangled parameter-efficient tuning
framework for subject-driven text-to-image generation. DisenBooth enables
generating new images that simultaneously preserve the subject identity and
conform to the text descriptions, by disentangling the embedding into an
identity-related and an identity-unrelated part. Specifically, DisenBooth is
based on the pretrained diffusion models and conducts finetuning in the
diffusion denoising process, where a shared identity embedding and an
image-specific identity-unrelated embedding are utilized jointly for denoising
each image. To make the two embeddings disentangled, two auxiliary objectives
are proposed. Additionally, to improve the finetuning efficiency, a
parameter-efficient finetuning strategy is adopted. Extensive experiments show
that our DisenBooth can faithfully learn well-disentangled identity-related and
identity-unrelated embeddings. With the shared identity embedding, DisenBooth
demonstrates superior subject-driven text-to-image generation ability.
Additionally, DisenBooth provides a more flexible and controllable framework
with different combinations of the disentangled embeddings.

提出 DisenBooth 框架，针对以往映射存在的问题，在加噪处理中对嵌入部分进行解缔。实验表明，DisenBooth 在以往的基础上，能够更好地学习与主题相关和不相关的嵌入，并且具有更强的主题驱动图像生成能力。