In this paper, we introduce PoseCrafter, a one-shot method for personalized
video generation following the control of flexible poses. Built upon Stable
Diffusion and ControlNet, we carefully design an inference process to produce
high-quality videos without the corresponding ground-truth frames. First, we
select an appropriate reference frame from the training video and invert it to
initialize all latent variables for generation. Then, we insert the
corresponding training pose into the target pose sequences to enhance
faithfulness through a trained temporal attention module. Furthermore, to
alleviate the face and hand degradation resulting from discrepancies between
poses of training videos and inference poses, we implement simple latent
editing through an affine transformation matrix involving facial and hand
landmarks. Extensive experiments on several datasets demonstrate that
PoseCrafter achieves superior results to baselines pre-trained on a vast
collection of videos under 8 commonly used metrics. Besides, PoseCrafter can
follow poses from different individuals or artificial edits and simultaneously
retain the human identity in an open-domain training video.

该论文介绍了 PoseCrafter，一种根据灵活姿势控制进行个性化视频生成的单次方法，通过稳定扩散和 ControlNet 构建，通过精心设计的推理过程，生成高质量视频而无需对应的真实帧。