Image inpainting, the process of restoring corrupted images, has seen
significant advancements with the advent of diffusion models (DMs). Despite
these advancements, current DM adaptations for inpainting, which involve
modifications to the sampling strategy or the development of
inpainting-specific DMs, frequently suffer from semantic inconsistencies and
reduced image quality. Addressing these challenges, our work introduces a novel
paradigm: the division of masked image features and noisy latent into separate
branches. This division dramatically diminishes the model's learning load,
facilitating a nuanced incorporation of essential masked image information in a
hierarchical fashion. Herein, we present BrushNet, a novel plug-and-play
dual-branch model engineered to embed pixel-level masked image features into
any pre-trained DM, guaranteeing coherent and enhanced image inpainting
outcomes. Additionally, we introduce BrushData and BrushBench to facilitate
segmentation-based inpainting training and performance assessment. Our
extensive experimental analysis demonstrates BrushNet's superior performance
over existing models across seven key metrics, including image quality, mask
region preservation, and textual coherence.

本研究提出了一种新的策略，将遮盖图像特征和噪声潜在特征分成独立分支，引入了 BrushNet 模型来将像素级遮盖图像特征嵌入任意预训练的扩散模型，以保证连贯和增强的图像修复结果，并通过多个指标评估实验证明了 BrushNet 在图像质量、遮盖区域保留和文本连贯性等七个关键度量上的优越性。