Despite the burst of innovative methods for controlling the diffusion process, effectively controlling image styles in text-to-image generation remains a challenging task. Many adapter-based methods impose image representation conditions on the denoising process to accomplish image control. However these conditions are not aligned with the word embedding space, leading to interference between image and text control conditions and the potential loss of semantic information from the text prompt. Addressing this issue involves two key challenges. Firstly, how to inject the style representation without compromising the effectiveness of text representation in control. Secondly, how to obtain the accurate style representation from a single reference image. To tackle these challenges, we introduce StyleTokenizer, a zero-shot style control image generation method that aligns style representation with text representation using a style tokenizer. This alignment effectively minimizes the impact on the effectiveness of text prompts. Furthermore, we collect a well-labeled style dataset named Style30k to train a style feature extractor capable of accurately representing style while excluding other content information. Experimental results demonstrate that our method fully grasps the style characteristics of the reference image, generating appealing images that are consistent with both the target image style and text prompt. The code and dataset are available at https://github.com/alipay/style-tokenizer.

本研究解决了在文本到图像生成中有效控制图像风格的难题。我们提出了StyleTokenizer，一种无须预先训练的风格控制生成方法，通过风格标记器将风格表示与文本表示对齐，从而减少对文本提示有效性的影响。实验结果表明，该方法能够准确捕捉参考图像的风格特征，同时生成与目标图像风格和文本提示一致的美观图像。

StyleTokenizer：通过单一实例定义图像风格以控制扩散模型