Generating customized content in videos has received increasing attention
recently. However, existing works primarily focus on customized text-to-video
generation for single subject, suffering from subject-missing and
attribute-binding problems when the video is expected to contain multiple
subjects. Furthermore, existing models struggle to assign the desired actions
to the corresponding subjects (action-binding problem), failing to achieve
satisfactory multi-subject generation performance. To tackle the problems, in
this paper, we propose DisenStudio, a novel framework that can generate
text-guided videos for customized multiple subjects, given few images for each
subject. Specifically, DisenStudio enhances a pretrained diffusion-based
text-to-video model with our proposed spatial-disentangled cross-attention
mechanism to associate each subject with the desired action. Then the model is
customized for the multiple subjects with the proposed motion-preserved
disentangled finetuning, which involves three tuning strategies: multi-subject
co-occurrence tuning, masked single-subject tuning, and multi-subject
motion-preserved tuning. The first two strategies guarantee the subject
occurrence and preserve their visual attributes, and the third strategy helps
the model maintain the temporal motion-generation ability when finetuning on
static images. We conduct extensive experiments to demonstrate our proposed
DisenStudio significantly outperforms existing methods in various metrics.
Additionally, we show that DisenStudio can be used as a powerful tool for
various controllable generation applications.

本文提出了一种名为 DisenStudio 的新框架，通过增强预训练的基于扩散的文本到视频模型，使用空间分离的交叉注意机制将每个主题与期望的动作相关联，并使用三种微调策略来定制多个主题。通过广泛的实验证明，DisenStudio 在各种指标上明显优于现有方法，并且可以作为各种可控生成应用的强大工具。

DisenStudio：定制化多主题文本到视频生成与解耦空间控制

DisenStudio: Customized Multi-subject Text-to-Video Generation with  Disentangled Spatial Control

3D facial animation is often produced by manipulating facial deformation
models (or rigs), that are traditionally parameterized by expression controls.
A key component that is usually overlooked is expression 'style', as in, how a
particular expression is performed. Although it is common to define a semantic
basis of expressions that characters can perform, most characters perform each
expression in their own style. To date, style is usually entangled with the
expression, and it is not possible to transfer the style of one character to
another when considering facial animation. We present a new face model, based
on a data-driven implicit neural physics model, that can be driven by both
expression and style separately. At the core, we present a framework for
learning implicit physics-based actuations for multiple subjects
simultaneously, trained on a few arbitrary performance capture sequences from a
small set of identities. Once trained, our method allows generalized
physics-based facial animation for any of the trained identities, extending to
unseen performances. Furthermore, it grants control over the animation style,
enabling style transfer from one character to another or blending styles of
different characters. Lastly, as a physics-based model, it is capable of
synthesizing physical effects, such as collision handling, setting our method
apart from conventional approaches.

基于数据驱动的隐式神经物理模型，我们提出了一种新的面部模型，可以分别通过表情和风格进行驱动，并且可以实现从一个角色到另一个角色的风格转移和不同角色之间的风格混合。

一种由表情和风格驱动的隐式物理人脸模型

An Implicit Physical Face Model Driven by Expression and Style

Synthesizing images with user-specified subjects has received growing
attention due to its practical applications. Despite the recent success in
single subject customization, existing algorithms suffer from high training
cost and low success rate along with increased number of subjects. Towards
controllable image synthesis with multiple subjects as the constraints, this
work studies how to efficiently represent a particular subject as well as how
to appropriately compose different subjects. We find that the text embedding
regarding the subject token already serves as a simple yet effective
representation that supports arbitrary combinations without any model tuning.
Through learning a residual on top of the base embedding, we manage to robustly
shift the raw subject to the customized subject given various text conditions.
We then propose to employ layout, a very abstract and easy-to-obtain prior, as
the spatial guidance for subject arrangement. By rectifying the activations in
the cross-attention map, the layout appoints and separates the location of
different subjects in the image, significantly alleviating the interference
across them. Both qualitative and quantitative experimental results demonstrate
our superiority over state-of-the-art alternatives under a variety of settings
for multi-subject customization.

研究在多个主题约束下如何高效地表示一个特定主题以及如何适当地组合不同主题，并提出了一种方案，该方案利用文本嵌入和布局来生成图像，并通过实验证明了其在多个主题的自定义方面具有优越性。