Recent progress in text-guided image inpainting, based on the unprecedented
success of text-to-image diffusion models, has led to exceptionally realistic
and visually plausible results. However, there is still significant potential
for improvement in current text-to-image inpainting models, particularly in
better aligning the inpainted area with user prompts and performing
high-resolution inpainting. Therefore, in this paper we introduce HD-Painter, a
completely training-free approach that accurately follows to prompts and
coherently scales to high-resolution image inpainting. To this end, we design
the Prompt-Aware Introverted Attention (PAIntA) layer enhancing self-attention
scores by prompt information and resulting in better text alignment
generations. To further improve the prompt coherence we introduce the
Reweighting Attention Score Guidance (RASG) mechanism seamlessly integrating a
post-hoc sampling strategy into general form of DDIM to prevent
out-of-distribution latent shifts. Moreover, HD-Painter allows extension to
larger scales by introducing a specialized super-resolution technique
customized for inpainting, enabling the completion of missing regions in images
of up to 2K resolution. Our experiments demonstrate that HD-Painter surpasses
existing state-of-the-art approaches qualitatively and quantitatively,
achieving an impressive generation accuracy improvement of 61.4% vs 51.9%. We
will make the codes publicly available at:
this https URL

最近的文本引导图像修复研究表明，HD-Painter 是一个完全无需训练的方法，可根据用户的提示进行高分辨率图像修复，并在文本对齐和生成准确性方面取得显著提升。

HD-Painter：具高分辨率和忠实预示文字引导的扩散模型图像修复

HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image  Inpainting with Diffusion Models

Text-guided image inpainting (TGII) aims to restore missing regions based on
a given text in a damaged image. Existing methods are based on a strong vision
encoder and a cross-modal fusion model to integrate cross-modal features.
However, these methods allocate most of the computation to visual encoding,
while light computation on modeling modality interactions. Moreover, they take
cross-modal fusion for depth features, which ignores a fine-grained alignment
between text and image. Recently, vision-language pre-trained models (VLPM),
encapsulating rich cross-modal alignment knowledge, have advanced in most
multimodal tasks. In this work, we propose a novel model for TGII by improving
cross-modal alignment (CMA). CMA model consists of a VLPM as a vision-language
encoder, an image generator and global-local discriminators. To explore
cross-modal alignment knowledge for image restoration, we introduce cross-modal
alignment distillation and in-sample distribution distillation. In addition, we
employ adversarial training to enhance the model to fill the missing region in
complicated structures effectively. Experiments are conducted on two popular
vision-language datasets. Results show that our model achieves state-of-the-art
performance compared with other strong competitors.

本研究提出了一种基于视听语言预训练模型和对跨模态对齐知识的改进的跨模态对齐模型的图像修复方法，通过在两个流行的视听语言数据集上的实验表明，该模型相对于其他强竞争对手的性能达到了最佳水平。