Based on recent advanced diffusion models, Text-to-image (T2I) generation models have demonstrated their capabilities in generating diverse and high-quality images. However, leveraging their potential for real-world content creation, particularly in providing users with precise control over the image generation result, poses a significant challenge. In this paper, we propose an innovative training-free pipeline that extends existing text-to-image generation models to incorporate a sketch as an additional condition. To generate new images with a layout and structure closely resembling the input sketch, we find that these core features of a sketch can be tracked with the cross-attention maps of diffusion models. We introduce latent optimization, a method that refines the noisy latent at each intermediate step of the generation process using cross-attention maps to ensure that the generated images closely adhere to the desired structure outlined in the reference sketch. Through latent optimization, our method enhances the fidelity and accuracy of image generation, offering users greater control and customization options in content creation.

本研究解决了现有文本到图像生成模型在用户对生成结果的精确控制方面的挑战。提出了一种新颖的无训练管道，利用草图作为附加条件，通过潜在优化方法提高生成图像与草图结构的准确性和保真度，从而增强用户在内容创作中的控制和定制选项。该方法有望在现实世界内容创作中提供更高的灵活性和质量。 

无训练的草图引导扩散模型与潜在优化