Generating high fidelity human video with specified identities has attracted
significant attention in the content generation community. However, existing
techniques struggle to strike a balance between training efficiency and
identity preservation, either requiring tedious case-by-case finetuning or
usually missing the identity details in video generation process. In this
study, we present ID-Animator, a zero-shot human-video generation approach that
can perform personalized video generation given single reference facial image
without further training. ID-Animator inherits existing diffusion-based video
generation backbones with a face adapter to encode the ID-relevant embeddings
from learnable facial latent queries. To facilitate the extraction of identity
information in video generation, we introduce an ID-oriented dataset
construction pipeline, which incorporates decoupled human attribute and action
captioning technique from a constructed facial image pool. Based on this
pipeline, a random face reference training method is further devised to
precisely capture the ID-relevant embeddings from reference images, thus
improving the fidelity and generalization capacity of our model for ID-specific
video generation. Extensive experiments demonstrate the superiority of
ID-Animator to generate personalized human videos over previous models.
Moreover, our method is highly compatible with popular pre-trained T2V models
like animatediff and various community backbone models, showing high
extendability in real-world applications for video generation where identity
preservation is highly desired. Our codes and checkpoints will be released at
this https URL

提供单个人脸参考图像，无需进一步训练，ID-Animator 是一种零样本的人类视频生成方法，具有个性化视频生成能力。该方法采用面部适配器从可学习的面部潜在查询中编码与身份相关的嵌入，通过 ID 导向的数据集构建技术和随机面部参考训练方法，提高模型在身份特定视频生成方面的准确性和泛化能力，实验结果表明 ID-Animator 比之前的模型更优秀，适用于对身份保护要求很高的实际视频生成应用。

ID-Animator：零槽位身份保持的人类视频生成

ID-Animator: Zero-Shot Identity-Preserving Human Video Generation

Text-driven content creation has evolved to be a transformative technique
that revolutionizes creativity. Here we study the task of text-driven human
video generation, where a video sequence is synthesized from texts describing
the appearance and motions of a target performer. Compared to general
text-driven video generation, human-centric video generation requires
maintaining the appearance of synthesized human while performing complex
motions. In this work, we present Text2Performer to generate vivid human videos
with articulated motions from texts. Text2Performer has two novel designs: 1)
decomposed human representation and 2) diffusion-based motion sampler. First,
we decompose the VQVAE latent space into human appearance and pose
representation in an unsupervised manner by utilizing the nature of human
videos. In this way, the appearance is well maintained along the generated
frames. Then, we propose continuous VQ-diffuser to sample a sequence of pose
embeddings. Unlike existing VQ-based methods that operate in the discrete
space, continuous VQ-diffuser directly outputs the continuous pose embeddings
for better motion modeling. Finally, motion-aware masking strategy is designed
to mask the pose embeddings spatial-temporally to enhance the temporal
coherence. Moreover, to facilitate the task of text-driven human video
generation, we contribute a Fashion-Text2Video dataset with manually annotated
action labels and text descriptions. Extensive experiments demonstrate that
Text2Performer generates high-quality human videos (up to 512x256 resolution)
with diverse appearances and flexible motions.

本文提出了 Text2Performer 算法，基于文本描述生成拥有精细动作的逼真人类视频，并提供了 Fashion-Text2Video 数据集用于辅助该任务。其中包括了两个新设计：将 VQVAE 潜空间表示细分为人物形象和姿势表示，并通过连续 VQ-diffuser 对姿势编码进行采样，得到更好的动作建模。该算法在多项实验中表现出成果丰硕。