Recent controllable generation approaches such as FreeControl and Diffusion
Self-guidance bring fine-grained spatial and appearance control to
text-to-image (T2I) diffusion models without training auxiliary modules.
However, these methods optimize the latent embedding for each type of score
function with longer diffusion steps, making the generation process
time-consuming and limiting their flexibility and use. This work presents
Ctrl-X, a simple framework for T2I diffusion controlling structure and
appearance without additional training or guidance. Ctrl-X designs feed-forward
structure control to enable the structure alignment with a structure image and
semantic-aware appearance transfer to facilitate the appearance transfer from a
user-input image. Extensive qualitative and quantitative experiments illustrate
the superior performance of Ctrl-X on various condition inputs and model
checkpoints. In particular, Ctrl-X supports novel structure and appearance
control with arbitrary condition images of any modality, exhibits superior
image quality and appearance transfer compared to existing works, and provides
instant plug-and-play functionality to any T2I and text-to-video (T2V)
diffusion model. See our project page for an overview of the results:
this https URL

Ctrl-X 是一个简单的 T2I 扩散（diffusion）控制结构和外观的框架，通过设计前馈结构控制以实现结构与结构图像的对齐以及语义感知外观转换，从而实现了对任意条件图像的全新结构和外观控制，展现了超过现有作品的图像质量和外观转换，并为任何 T2I 和 T2V 扩散模型提供即插即用的功能。