Diffusion models have achieved remarkable success in generating realistic
images but suffer from generating accurate human hands, such as incorrect
finger counts or irregular shapes. This difficulty arises from the complex task
of learning the physical structure and pose of hands from training images,
which involves extensive deformations and occlusions. For correct hand
generation, our paper introduces a lightweight post-processing solution called
$\textbf{HandRefiner}$. HandRefiner employs a conditional inpainting approach
to rectify malformed hands while leaving other parts of the image untouched. We
leverage the hand mesh reconstruction model that consistently adheres to the
correct number of fingers and hand shape, while also being capable of fitting
the desired hand pose in the generated image. Given a generated failed image
due to malformed hands, we utilize ControlNet modules to re-inject such correct
hand information. Additionally, we uncover a phase transition phenomenon within
ControlNet as we vary the control strength. It enables us to take advantage of
more readily available synthetic data without suffering from the domain gap
between realistic and synthetic hands. Experiments demonstrate that HandRefiner
can significantly improve the generation quality quantitatively and
qualitatively. The code is available at
this https URL .

我们的研究引入了一种轻量级的后处理方案 HandRefiner，通过一种条件修复方法来纠正生成图像中畸形的手部，同时保持图像其他部分的原貌。我们利用手部网格重建模型，确保生成图像中手指数量和手部形状正确，同时可以适应所期望的手部姿态。通过利用控制网络模块将正确的手部信息重新注入到生成失败的图像中，我们改进了手部生成的质量。实验证明，HandRefiner 能够在定量和定性方面显著提高生成质量。

HandRefiner: 基于扩散的条件修复技术对生成图像中的畸形手进行修复

HandRefiner: Refining Malformed Hands in Generated Images by  Diffusion-based Conditional Inpainting

Sketch-based image manipulation is an interactive image editing task to
modify an image based on input sketches from users. Existing methods typically
formulate this task as a conditional inpainting problem, which requires users
to draw an extra mask indicating the region to modify in addition to sketches.
The masked regions are regarded as holes and filled by an inpainting model
conditioned on the sketch. With this formulation, paired training data can be
easily obtained by randomly creating masks and extracting edges or contours.
Although this setup simplifies data preparation and model design, it
complicates user interaction and discards useful information in masked regions.
To this end, we investigate a new paradigm of sketch-based image manipulation:
mask-free local image manipulation, which only requires sketch inputs from
users and utilizes the entire original image. Given an image and sketch, our
model automatically predicts the target modification region and encodes it into
a structure agnostic style vector. A generator then synthesizes the new image
content based on the style vector and sketch. The manipulated image is finally
produced by blending the generator output into the modification region of the
original image. Our model can be trained in a self-supervised fashion by
learning the reconstruction of an image region from the style vector and
sketch. The proposed method offers simpler and more intuitive user workflows
for sketch-based image manipulation and provides better results than previous
approaches. More results, code and interactive demo will be available at
https://zengxianyu.github.io/sketchedit.

基于草图的图像操作旨在通过用户的输入草图修改图像。本文提出一种新的草图驱动图像操作方法，名为无蒙版本地图像操作，只需要用户提供草图输入，并利用整个原始图像。

SketchEdit: 用局部草稿实现无蒙版图像局部处理

SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches

Single image generative models perform synthesis and manipulation tasks by
capturing the distribution of patches within a single image. The classical (pre
Deep Learning) prevailing approaches for these tasks are based on an
optimization process that maximizes patch similarity between the input and
generated output. Recently, however, Single Image GANs were introduced both as
a superior solution for such manipulation tasks, but also for remarkable novel
generative tasks. Despite their impressiveness, single image GANs require long
training time (usually hours) for each image and each task. They often suffer
from artifacts and are prone to optimization issues such as mode collapse. In
this paper, we show that all of these tasks can be performed without any
training, within several seconds, in a unified, surprisingly simple framework.
We revisit and cast the "good-old" patch-based methods into a novel
optimization-free framework. We start with an initial coarse guess, and then
simply refine the details coarse-to-fine using patch-nearest-neighbor search.
This allows generating random novel images better and much faster than GANs. We
further demonstrate a wide range of applications, such as image editing and
reshuffling, retargeting to different sizes, structural analogies, image
collage and a newly introduced task of conditional inpainting. Not only is our
method faster ($\times 10^3$-$\times 10^4$ than a GAN), it produces superior
results (confirmed by quantitative and qualitative evaluation), less artifacts
and more realistic global structure than any of the previous approaches
(whether GAN-based or classical patch-based).

本文提出了一种不需要训练的基于补丁的优化框架，比单个图像 GAN 方法快 1000~10000 倍，并产生比任何以前的方法（无论是基于 GAN 还是基于经典补丁的方法）更优越的结果和更真实的整体结构，应用广泛，如图像编辑和重塑大小。