Generating text-editable and pose-controllable character videos have an
imperious demand in creating various digital human. Nevertheless, this task has
been restricted by the absence of a comprehensive dataset featuring paired
video-pose captions and the generative prior models for videos. In this work,
we design a novel two-stage training scheme that can utilize easily obtained
datasets (i.e.,image pose pair and pose-free video) and the pre-trained
text-to-image (T2I) model to obtain the pose-controllable character videos.
Specifically, in the first stage, only the keypoint-image pairs are used only
for a controllable text-to-image generation. We learn a zero-initialized
convolu- tional encoder to encode the pose information. In the second stage, we
finetune the motion of the above network via a pose-free video dataset by
adding the learnable temporal self-attention and reformed cross-frame
self-attention blocks. Powered by our new designs, our method successfully
generates continuously pose-controllable character videos while keeps the
editing and concept composition ability of the pre-trained T2I model. The code
and models will be made publicly available.

本研究提出一种新的双阶段训练方法，可利用易获得的数据集和预训练的 text-to-image（T2I）模型，生成可编辑和姿态可控的字符视频。这种方法成功地生成了连续的姿态可控字符视频，同时保持了预训练 T2I 模型的编辑和概念构成能力。