Image style transfer aims to imbue digital imagery with the distinctive
attributes of style targets, such as colors, brushstrokes, shapes, whilst
concurrently preserving the semantic integrity of the content. Despite the
advancements in arbitrary style transfer methods, a prevalent challenge remains
the delicate equilibrium between content semantics and style attributes. Recent
developments in large-scale text-to-image diffusion models have heralded
unprecedented synthesis capabilities, albeit at the expense of relying on
extensive and often imprecise textual descriptions to delineate artistic
styles. Addressing these limitations, this paper introduces DiffStyler, a novel
approach that facilitates efficient and precise arbitrary image style transfer.
DiffStyler lies the utilization of a text-to-image Stable Diffusion model-based
LoRA to encapsulate the essence of style targets. This approach, coupled with
strategic cross-LoRA feature and attention injection, guides the style transfer
process. The foundation of our methodology is rooted in the observation that
LoRA maintains the spatial feature consistency of UNet, a discovery that
further inspired the development of a mask-wise style transfer technique. This
technique employs masks extracted through a pre-trained FastSAM model,
utilizing mask prompts to facilitate feature fusion during the denoising
process, thereby enabling localized style transfer that preserves the original
image's unaffected regions. Moreover, our approach accommodates multiple style
targets through the use of corresponding masks. Through extensive
experimentation, we demonstrate that DiffStyler surpasses previous methods in
achieving a more harmonious balance between content preservation and style
integration.

DiffStyler 是一种新方法，通过利用基于 LoRA 的文本到图像扩散模型来实现高效且精确的任意图像风格转移，该方法结合了跨 LoRA 特征和注意注入的策略，旨在在内容保持和风格整合之间取得更和谐的平衡。