Recently, the multimedia community has witnessed the rise of diffusion models
trained on large-scale multi-modal data for visual content creation,
particularly in the field of text-to-image generation. In this paper, we
propose a new task for ``stylizing'' text-to-image models, namely text-driven
stylized image generation, that further enhances editability in content
creation. Given input text prompt and style image, this task aims to produce
stylized images which are both semantically relevant to input text prompt and
meanwhile aligned with the style image in style. To achieve this, we present a
new diffusion model (ControlStyle) via upgrading a pre-trained text-to-image
model with a trainable modulation network enabling more conditions of text
prompts and style images. Moreover, diffusion style and content regularizations
are simultaneously introduced to facilitate the learning of this modulation
network with these diffusion priors, pursuing high-quality stylized
text-to-image generation. Extensive experiments demonstrate the effectiveness
of our ControlStyle in producing more visually pleasing and artistic results,
surpassing a simple combination of text-to-image model and conventional style
transfer techniques.

提出了一种新的任务，文本驱动的风格化图像生成，以进一步增强内容创造中的可编辑性，通过升级经过训练的文本到图像模型与可训练的调制网络，同时引入扩散样式和内容正则化，实现了高质量的风格化文本到图像生成。

控制样式：使用扩散先验生成基于文本的风格化图像

ControlStyle: Text-Driven Stylized Image Generation Using Diffusion  Priors

This paper presents a LoRA-free method for stylized image generation that
takes a text prompt and style reference images as inputs and produces an output
image in a single pass. Unlike existing methods that rely on training a
separate LoRA for each style, our method can adapt to various styles with a
unified model. However, this poses two challenges: 1) the prompt loses
controllability over the generated content, and 2) the output image inherits
both the semantic and style features of the style reference image, compromising
its content fidelity. To address these challenges, we introduce StyleAdapter, a
model that comprises two components: a two-path cross-attention module (TPCA)
and three decoupling strategies. These components enable our model to process
the prompt and style reference features separately and reduce the strong
coupling between the semantic and style information in the style references.
StyleAdapter can generate high-quality images that match the content of the
prompts and adopt the style of the references (even for unseen styles) in a
single pass, which is more flexible and efficient than previous methods.
Experiments have been conducted to demonstrate the superiority of our method
over previous works.

该研究提出了一种无需 LoRA 的方法，用于风格化图像生成，通过使用文本提示和风格参考图像作为输入，以单次传递生成输出图像。使用统一模型，该方法能够适应多种风格，但面临两个挑战：1）提示在生成内容上失去可控性，2）输出图像同时继承了风格参考图像的语义和风格特征，从而损害其内容的保真度。为解决这些挑战，作者提出了 StyleAdapter，该模型由两个组件组成：双路径交叉注意力模块（TPCA）和三个解耦策略。这些组件使模型能够分别处理提示和风格参考特征，并减少风格参考中语义和风格信息之间的强耦合。StyleAdapter 能够以单次传递生成与提示内容匹配且采用参考的风格的高质量图像，相比以前的方法更灵活高效。实验证明了我们方法的优越性。