Diffusion models have achieved remarkable success in generating realistic
images but suffer from generating accurate human hands, such as incorrect
finger counts or irregular shapes. This difficulty arises from the complex task
of learning the physical structure and pose of hands from training images,
which involves extensive deformations and occlusions. For correct hand
generation, our paper introduces a lightweight post-processing solution called
$\textbf{HandRefiner}$. HandRefiner employs a conditional inpainting approach
to rectify malformed hands while leaving other parts of the image untouched. We
leverage the hand mesh reconstruction model that consistently adheres to the
correct number of fingers and hand shape, while also being capable of fitting
the desired hand pose in the generated image. Given a generated failed image
due to malformed hands, we utilize ControlNet modules to re-inject such correct
hand information. Additionally, we uncover a phase transition phenomenon within
ControlNet as we vary the control strength. It enables us to take advantage of
more readily available synthetic data without suffering from the domain gap
between realistic and synthetic hands. Experiments demonstrate that HandRefiner
can significantly improve the generation quality quantitatively and
qualitatively. The code is available at
this https URL .

我们的研究引入了一种轻量级的后处理方案 HandRefiner，通过一种条件修复方法来纠正生成图像中畸形的手部，同时保持图像其他部分的原貌。我们利用手部网格重建模型，确保生成图像中手指数量和手部形状正确，同时可以适应所期望的手部姿态。通过利用控制网络模块将正确的手部信息重新注入到生成失败的图像中，我们改进了手部生成的质量。实验证明，HandRefiner 能够在定量和定性方面显著提高生成质量。

HandRefiner: 基于扩散的条件修复技术对生成图像中的畸形手进行修复

HandRefiner: Refining Malformed Hands in Generated Images by  Diffusion-based Conditional Inpainting

Text-to-image diffusion models can synthesize a large variety of concepts in
new compositions and scenarios. However, they still struggle with generating
uncommon concepts, rare unusual combinations, or structured concepts like hand
palms. Their limitation is partly due to the long-tail nature of their training
data: web-crawled data sets are strongly unbalanced, causing models to
under-represent concepts from the tail of the distribution. Here we
characterize the effect of unbalanced training data on text-to-image models and
offer a remedy. We show that rare concepts can be correctly generated by
carefully selecting suitable generation seeds in the noise space, a technique
that we call SeedSelect. SeedSelect is efficient and does not require
retraining the diffusion model. We evaluate the benefit of SeedSelect on a
series of problems. First, in few-shot semantic data augmentation, where we
generate semantically correct images for few-shot and long-tail benchmarks. We
show classification improvement on all classes, both from the head and tail of
the training data of diffusion models. We further evaluate SeedSelect on
correcting images of hands, a well-known pitfall of current diffusion models,
and show that it improves hand generation substantially.

研究了长尾训练数据对于文本到图像模型的影响，并提出了 SeedSelect 技术，通过在噪声空间中选择适当的初始种子，可以有效地生成罕见的图像概念，提高了模型效果，特别是在手部图像生成方面的表现得到了明显改善。