Controllable text-to-image (T2I) diffusion models have shown impressive
performance in generating high-quality visual content through the incorporation
of various conditions. Current methods, however, exhibit limited performance
when guided by skeleton human poses, especially in complex pose conditions such
as side or rear perspectives of human figures. To address this issue, we
present Stable-Pose, a novel adapter model that introduces a coarse-to-fine
attention masking strategy into a vision Transformer (ViT) to gain accurate
pose guidance for T2I models. Stable-Pose is designed to adeptly handle pose
conditions within pre-trained Stable Diffusion, providing a refined and
efficient way of aligning pose representation during image synthesis. We
leverage the query-key self-attention mechanism of ViTs to explore the
interconnections among different anatomical parts in human pose skeletons.
Masked pose images are used to smoothly refine the attention maps based on
target pose-related features in a hierarchical manner, transitioning from
coarse to fine levels. Additionally, our loss function is formulated to
allocate increased emphasis to the pose region, thereby augmenting the model's
precision in capturing intricate pose details. We assessed the performance of
Stable-Pose across five public datasets under a wide range of indoor and
outdoor human pose scenarios. Stable-Pose achieved an AP score of 57.1 in the
LAION-Human dataset, marking around 13% improvement over the established
technique ControlNet. The project link and code is available at
this https URL

通过引入粗糙到精细的注意掩蔽策略到视觉 Transformer (ViT) 中，我们提出了一个新的适配器模型 Stable-Pose，以在生成图像时获得准确的姿势指导，并通过层级方式从粗糙到精细的过渡来提供一种对齐姿势表示的优化方式。