Stylized Text-to-Image Generation (STIG) aims to generate images based on
text prompts and style reference images. We in this paper propose a novel
framework dubbed as StyleMaster for this task by leveraging pretrained Stable
Diffusion (SD), which tries to solve the previous problems such as insufficient
style and inconsistent semantics. The enhancement lies in two novel module,
namely multi-source style embedder and dynamic attention adapter. In order to
provide SD with better style embeddings, we propose the multi-source style
embedder considers both global and local level visual information along with
textual one, which provide both complementary style-related and
semantic-related knowledge. Additionally, aiming for better balance between the
adaptor capacity and semantic control, the proposed dynamic attention adapter
is applied to the diffusion UNet in which adaptation weights are dynamically
calculated based on the style embeddings. Two objective functions are
introduced to optimize the model together with denoising loss, which can
further enhance semantic and style consistency. Extensive experiments
demonstrate the superiority of StyleMaster over existing methods, rendering
images with variable target styles while successfully maintaining the semantic
information from the text prompts.

Stylized Text-to-Image Generation paper proposes StyleMaster, a framework utilizing pretrained Stable Diffusion for generating images from text prompts, overcoming previous issues with insufficient style and inconsistent semantics. It introduces a multi-source style embedder and dynamic attention adapter to provide improved style embeddings and adaptability, and evaluates the model using objective functions and denoising loss, demonstrating its superior performance in achieving variable target styles while maintaining semantic information.

StyleMaster：面向灵活样式化图像生成的扩散模型

StyleMaster: Towards Flexible Stylized Image Generation with Diffusion  Models

Stylized text-to-image generation focuses on creating images from textual
descriptions while adhering to a style specified by a few reference images.
However, subtle style variations within different reference images can hinder
the model from accurately learning the target style. In this paper, we propose
InstaStyle, a novel approach that excels in generating high-fidelity stylized
images with only a single reference image. Our approach is based on the finding
that the inversion noise from a stylized reference image inherently carries the
style signal, as evidenced by their non-zero signal-to-noise ratio. We employ
DDIM inversion to extract this noise from the reference image and leverage a
diffusion model to generate new stylized images from the ``style" noise.
Additionally, the inherent ambiguity and bias of textual prompts impede the
precise conveying of style. To address this, we introduce a learnable style
token via prompt refinement, which enhances the accuracy of the style
description for the reference image. Qualitative and quantitative experimental
results demonstrate that InstaStyle achieves superior performance compared to
current benchmarks. Furthermore, our approach also showcases its capability in
the creative task of style combination with mixed inversion noise.

InstaStyle 是一种新方法，通过从单个参考图像中提取风格噪声，并利用扩散模型生成新的高保真度风格化图像，能够在创作任务和样式组合方面提供卓越的性能。