Diffusion-based models have demonstrated impressive capabilities for
text-to-image generation and are expected for personalized applications of
subject-driven generation, which require the generation of customized concepts
with one or a few reference images. However, existing methods based on
fine-tuning fail to balance the trade-off between subject learning and the
maintenance of the generation capabilities of pretrained models. Moreover,
other methods that utilize additional image encoders tend to lose important
details of the subject due to encoding compression. To address these
challenges, we propose DreamTurner, a novel method that injects reference
information from coarse to fine to achieve subject-driven image generation more
effectively. DreamTurner introduces a subject-encoder for coarse subject
identity preservation, where the compressed general subject features are
introduced through an attention layer before visual-text cross-attention. We
then modify the self-attention layers within pretrained text-to-image models to
self-subject-attention layers to refine the details of the target subject. The
generated image queries detailed features from both the reference image and
itself in self-subject-attention. It is worth emphasizing that
self-subject-attention is an effective, elegant, and training-free method for
maintaining the detailed features of customized subjects and can serve as a
plug-and-play solution during inference. Finally, with additional
subject-driven fine-tuning, DreamTurner achieves remarkable performance in
subject-driven image generation, which can be controlled by a text or other
conditions such as pose. For further details, please visit the project page at
this https URL

DreamTurner 是一种新方法，通过将粗略主题特征逐渐注入到细节中，有效地实现主题驱动的图像生成，其中引入了主题编码器以保留粗略主题特征并使用自主题注意力层来提炼目标主题的细节特征。