Recently, text-to-image denoising diffusion probabilistic models (DDPMs) have
demonstrated impressive image generation capabilities and have also been
successfully applied to image inpainting. However, in practice, users often
require more control over the inpainting process beyond textual guidance,
especially when they want to composite objects with customized appearance,
color, shape, and layout. Unfortunately, existing diffusion-based inpainting
methods are limited to single-modal guidance and require task-specific
training, hindering their cross-modal scalability. To address these
limitations, we propose Uni-paint, a unified framework for multimodal
inpainting that offers various modes of guidance, including unconditional,
text-driven, stroke-driven, exemplar-driven inpainting, as well as a
combination of these modes. Furthermore, our Uni-paint is based on pretrained
Stable Diffusion and does not require task-specific training on specific
datasets, enabling few-shot generalizability to customized images. We have
conducted extensive qualitative and quantitative evaluations that show our
approach achieves comparable results to existing single-modal methods while
offering multimodal inpainting capabilities not available in other methods.
Code will be available at this https URL

提出了一种名为 Uni-paint 的多模态修复统一框架，基于 Stable Diffusion 预训练模型，支持无条件、文本驱动、笔画驱动、示例驱动等多种修复模态，无需特定数据集的任务特定训练，通过广泛的定性和定量评估表明，该方法在保持与现有单模态方法具有可比性的基础上，提供了其他方法中不可用的多模态修复能力。