Diffusion models are now the undisputed state-of-the-art for image generation
and image restoration. However, they require large amounts of computational
power for training and inference. In this paper, we propose lightweight
diffusion models for image inpainting that can be trained on a single image, or
a few images. We show that our approach competes with large state-of-the-art
models in specific cases. We also show that training a model on a single image
is particularly relevant for image acquisition modality that differ from the
RGB images of standard learning databases. We show results in three different
contexts: texture images, line drawing images, and materials BRDF, for which we
achieve state-of-the-art results in terms of realism, with a computational load
that is greatly reduced compared to concurrent methods.

本文提出了一种用于图像修复的轻量级扩散模型，可以在单幅图像或几幅图像上进行训练，结果表明我们的方法在某些情况下可以与大型先进模型竞争，特别适用于与标准学习数据库中的 RGB 图像不同的图像获取模态，并在纹理图像、线条图像和材料 BRDF 等三个不同情境中取得了逼真度方面的最新成果，与相应方法相比计算负荷大大减少。

基于扩散的图像修复与内部学习

Diffusion-based image inpainting with internal learning

In this paper, we present DeepSIM, a generative model for conditional image
manipulation based on a single image. We find that extensive augmentation is
key for enabling single image training, and incorporate the use of
thin-plate-spline (TPS) as an effective augmentation. Our network learns to map
between a primitive representation of the image to the image itself. The choice
of a primitive representation has an impact on the ease and expressiveness of
the manipulations and can be automatic (e.g. edges), manual (e.g. segmentation)
or hybrid such as edges on top of segmentations. At manipulation time, our
generator allows for making complex image changes by modifying the primitive
input representation and mapping it through the network. Our method is shown to
achieve remarkable performance on image manipulation tasks.

本文介绍了 DeepSIM ，这是一种基于单个图像的生成模型，用于条件图像操作。我们发现广泛的数据增强是实现单个图像训练的关键，并整合了 thin-plate-spline 作为有效的数据增强。我们的网络学会了将图像的原始表示映射到图像本身。基于不同的原始表示形式进行的操作对易用性和表达能力都产生重大影响，可以是自动的（例如图像边缘），手动的（例如分割）或混合的。在操作时间，通过修改原始输入表示并通过网络映射它，我们的生成器允许进行复杂的图像更改。我们的方法在图像操作任务上表现出了出众的性能。