Identity-consistent video generation seeks to synthesize videos that are
guided by both textual prompts and reference images of entities. Current
approaches typically utilize cross-attention layers to integrate the appearance
of the entity, which predominantly captures semantic attributes, resulting in
compromised fidelity of entities. Moreover, these methods necessitate iterative
fine-tuning for each new entity encountered, thereby limiting their
applicability. To address these challenges, we introduce VideoAssembler, a
novel end-to-end framework for identity-consistent video generation that can
conduct inference directly when encountering new entities. VideoAssembler is
adept at producing videos that are not only flexible with respect to the input
reference entities but also responsive to textual conditions. Additionally, by
modulating the quantity of input images for the entity, VideoAssembler enables
the execution of tasks ranging from image-to-video generation to sophisticated
video editing. VideoAssembler comprises two principal components: the Reference
Entity Pyramid (REP) encoder and the Entity-Prompt Attention Fusion (EPAF)
module. The REP encoder is designed to infuse comprehensive appearance details
into the denoising stages of the stable diffusion model. Concurrently, the EPAF
module is utilized to integrate text-aligned features effectively. Furthermore,
to mitigate the challenge of scarce data, we present a methodology for the
preprocessing of training data. Our evaluation of the VideoAssembler framework
on the UCF-101, MSR-VTT, and DAVIS datasets indicates that it achieves good
performances in both quantitative and qualitative analyses (346.84 in FVD and
48.01 in IS on UCF-101). Our project page is at
this https URL

VideoAssembler 是一种新颖的端到端框架，用于保持一致的身份生成视频，并可直接进行推理，以适应新实体的要求。它能够在灵活处理输入参考实体的同时，对文本条件作出响应，还可以根据输入图像的数量进行任务执行，从图像到视频生成到复杂的视频编辑。