We present a method for multi-concept customization of pretrained
text-to-video (T2V) models. Intuitively, the multi-concept customized video can
be derived from the (non-linear) intersection of the video manifolds of the
individual concepts, which is not straightforward to find. We hypothesize that
sequential and controlled walking towards the intersection of the video
manifolds, directed by text prompting, leads to the solution. To do so, we
generate the various concepts and their corresponding interactions,
sequentially, in an autoregressive manner. Our method can generate videos of
multiple custom concepts (subjects, action and background) such as a teddy bear
running towards a brown teapot, a dog playing violin and a teddy bear swimming
in the ocean. We quantitatively evaluate our method using videoCLIP and DINO
scores, in addition to human evaluation. Videos for results presented in this
paper can be found at this https URL

我们提出了一种多概念定制预训练文本到视频模型的方法，通过自动回归的方式生成多个定制概念的视频，并使用 videoCLIP、DINO 分数以及人工评估进行定量评估。

自回归生成的多概念视频定制的文本引导

Text Prompting for Multi-Concept Video Customization by Autoregressive  Generation

This paper introduces MultiBooth, a novel and efficient technique for
multi-concept customization in image generation from text. Despite the
significant advancements in customized generation methods, particularly with
the success of diffusion models, existing methods often struggle with
multi-concept scenarios due to low concept fidelity and high inference cost.
MultiBooth addresses these issues by dividing the multi-concept generation
process into two phases: a single-concept learning phase and a multi-concept
integration phase. During the single-concept learning phase, we employ a
multi-modal image encoder and an efficient concept encoding technique to learn
a concise and discriminative representation for each concept. In the
multi-concept integration phase, we use bounding boxes to define the generation
area for each concept within the cross-attention map. This method enables the
creation of individual concepts within their specified regions, thereby
facilitating the formation of multi-concept images. This strategy not only
improves concept fidelity but also reduces additional inference cost.
MultiBooth surpasses various baselines in both qualitative and quantitative
evaluations, showcasing its superior performance and computational efficiency.
Project Page: this https URL

介绍了一种名为 MultiBooth 的新颖高效的图像生成技术，该技术可在文本中进行多概念定制生成；通过将多概念生成过程分为单概念学习阶段和多概念整合阶段，利用多模态图像编码和高效的概念编码技术提高了概念准确性并降低了推断成本。

MultiBooth: 面向图像中全部文本概念生成

MultiBooth: Towards Generating All Your Concepts in an Image from Text

Customization generation techniques have significantly advanced the synthesis
of specific concepts across varied contexts. Multi-concept customization
emerges as the challenging task within this domain. Existing approaches often
rely on training a Low-Rank Adaptations (LoRA) fusion matrix of multiple LoRA
to merge various concepts into a single image. However, we identify this
straightforward method faces two major challenges: 1) concept confusion, which
occurs when the model cannot preserve distinct individual characteristics, and
2) concept vanishing, where the model fails to generate the intended subjects.
To address these issues, we introduce LoRA-Composer, a training-free framework
designed for seamlessly integrating multiple LoRAs, thereby enhancing the
harmony among different concepts within generated images. LoRA-Composer
addresses concept vanishing through Concept Injection Constraints, enhancing
concept visibility via an expanded cross-attention mechanism. To combat concept
confusion, Concept Isolation Constraints are introduced, refining the
self-attention computation. Furthermore, Latent Re-initialization is proposed
to effectively stimulate concept-specific latent within designated regions. Our
extensive testing showcases a notable enhancement in LoRA-Composer's
performance compared to standard baselines, especially when eliminating the
image-based conditions like canny edge or pose estimations. Code is released at
this https URL

自定义生成技术在不同背景下合成特定概念的技术已经取得了显著进展。多概念自定义是该领域内的一项具有挑战性的任务。现有方法往往依赖于训练多个低秩适应矩阵（LoRA）的融合矩阵，将各种概念合并到单个图像中。然而，我们发现这种直接方法面临两个主要挑战：1）概念混淆，即模型无法保留独特的个体特征，2）概念消失，即模型无法生成预期的主题。为了解决这些问题，我们引入了一种名为 LoRA-Composer 的无需训练的框架，用于无缝集成多个 LoRA，从而增强生成图像中不同概念之间的和谐。LoRA-Composer 通过概念注入约束来解决概念消失问题，通过扩展交叉注意机制增强概念可见性。为了对抗概念混淆，引入了概念隔离约束，对自注意力计算进行了改进。此外，提出了潜在重新初始化方法来有效激发指定区域内的特定概念潜在性。我们的广泛测试显示，与标准基线方法相比，LoRA-Composer 的性能有明显的提升，特别是在消除基于图像的条件（如边缘检测或姿势估计）时。代码发布在此 https URL