Despite the success in large-scale text-to-image generation and
text-conditioned image editing, existing methods still struggle to produce
consistent generation and editing results. For example, generation approaches
usually fail to synthesize multiple images of the same objects/characters but
with different views or poses. Meanwhile, existing editing methods either fail
to achieve effective complex non-rigid editing while maintaining the overall
textures and identity, or require time-consuming fine-tuning to capture the
image-specific appearance. In this paper, we develop MasaCtrl, a tuning-free
method to achieve consistent image generation and complex non-rigid image
editing simultaneously. Specifically, MasaCtrl converts existing self-attention
in diffusion models into mutual self-attention, so that it can query correlated
local contents and textures from source images for consistency. To further
alleviate the query confusion between foreground and background, we propose a
mask-guided mutual self-attention strategy, where the mask can be easily
extracted from the cross-attention maps. Extensive experiments show that the
proposed MasaCtrl can produce impressive results in both consistent image
generation and complex non-rigid real image editing.

提出了一种名为 MasaCtrl 的调整无需微调的方法，能够同时实现一致的图像生成和复杂的非刚性图像编辑。具体来说，该方法将扩散模型中的自我注意力转换为相互自我注意力，以便对源图像进行一致的局部内容和纹理查询。同时，还提出了一种掩码引导的相互关注策略，可减轻前景和背景之间的查询混乱。大量实验证明，该方法可以在一致的图像生成和复杂的非刚性真实图像编辑方面产生令人印象深刻的结果。