Personalizing a large-scale pretrained Text-to-Image (T2I) diffusion model is
challenging as it typically struggles to make an appropriate trade-off between
its training data distribution and the target distribution, i.e., learning a
novel concept with only a few target images to achieve personalization
(aligning with the personalized target) while preserving text editability
(aligning with diverse text prompts). In this paper, we propose PaRa, an
effective and efficient Parameter Rank Reduction approach for T2I model
personalization by explicitly controlling the rank of the diffusion model
parameters to restrict its initial diverse generation space into a small and
well-balanced target space. Our design is motivated by the fact that taming a
T2I model toward a novel concept such as a specific art style implies a small
generation space. To this end, by reducing the rank of model parameters during
finetuning, we can effectively constrain the space of the denoising sampling
trajectories towards the target. With comprehensive experiments, we show that
PaRa achieves great advantages over existing finetuning approaches on
single/multi-subject generation as well as single-image editing. Notably,
compared to the prevailing fine-tuning technique LoRA, PaRa achieves better
parameter efficiency (2x fewer learnable parameters) and much better target
image alignment.

PaRa 是一种用于 T2I 模型个性化的参数等级减少方法，通过显式控制扩散模型参数的等级来限制其初始的多样化生成空间为一个小而平衡的目标空间，通过全面实验证明，PaRa 在单 / 多主题生成以及单图像编辑方面相比现有的微调方法具有更好的参数效率（2 倍更少的可学习参数）和更好的目标图像对齐效果。

PaRa: 個性化文本到圖像擴散通過參數等級降低

PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction

Personalized text-to-image (T2I) synthesis based on diffusion models has
attracted significant attention in recent research. However, existing methods
primarily concentrate on customizing subjects or styles, neglecting the
exploration of global geometry. In this study, we propose an approach that
focuses on the customization of 360-degree panoramas, which inherently possess
global geometric properties, using a T2I diffusion model. To achieve this, we
curate a paired image-text dataset specifically designed for the task and
subsequently employ it to fine-tune a pre-trained T2I diffusion model with
LoRA. Nevertheless, the fine-tuned model alone does not ensure the continuity
between the leftmost and rightmost sides of the synthesized images, a crucial
characteristic of 360-degree panoramas. To address this issue, we propose a
method called StitchDiffusion. Specifically, we perform pre-denoising
operations twice at each time step of the denoising process on the stitch block
consisting of the leftmost and rightmost image regions. Furthermore, a global
cropping is adopted to synthesize seamless 360-degree panoramas. Experimental
results demonstrate the effectiveness of our customized model combined with the
proposed StitchDiffusion in generating high-quality 360-degree panoramic
images. Moreover, our customized model exhibits exceptional generalization
ability in producing scenes unseen in the fine-tuning dataset. Code is
available at this https URL

基于扩散模型的个性化文本到图像（T2I）合成已经引起了最近研究的广泛关注，然而现有的方法主要集中在定制主题或风格上，忽视了全局几何的探索。本研究提出了一种集中于 360 度全景定制的方法，使用 T2I 扩散模型，它在本质上具有全局几何特性，通过精心制作的图像 - 文本配对数据集进行微调，最终采用 LoRA 进行训练。实验证明了我们定制的模型与所提出的 StitchDiffusion 相结合在生成高质量的 360 度全景图像方面的有效性。此外，我们的定制模型在产生未在微调数据集中见过的场景方面表现出卓越的泛化能力。

通过文本与图像扩散模型个性化定制 360 度全景图

Customizing 360-Degree Panoramas through Text-to-Image Diffusion Models

The rising demand for creating lifelike avatars in the digital realm has led
to an increased need for generating high-quality human videos guided by textual
descriptions and poses. We propose Dancing Avatar, designed to fabricate human
motion videos driven by poses and textual cues. Our approach employs a
pretrained T2I diffusion model to generate each video frame in an
autoregressive fashion. The crux of innovation lies in our adept utilization of
the T2I diffusion model for producing video frames successively while
preserving contextual relevance. We surmount the hurdles posed by maintaining
human character and clothing consistency across varying poses, along with
upholding the background's continuity amidst diverse human movements. To ensure
consistent human appearances across the entire video, we devise an intra-frame
alignment module. This module assimilates text-guided synthesized human
character knowledge into the pretrained T2I diffusion model, synergizing
insights from ChatGPT. For preserving background continuity, we put forth a
background alignment pipeline, amalgamating insights from segment anything and
image inpainting techniques. Furthermore, we propose an inter-frame alignment
module that draws inspiration from an auto-regressive pipeline to augment
temporal consistency between adjacent frames, where the preceding frame guides
the synthesis process of the current frame. Comparisons with state-of-the-art
methods demonstrate that Dancing Avatar exhibits the capacity to generate human
videos with markedly superior quality, both in terms of human and background
fidelity, as well as temporal coherence compared to existing state-of-the-art
approaches.

提出了一种名为 “Dancing Avatar” 的方法，通过文本和姿势驱动，利用训练良好的 T2I 扩散模型生成逐帧人体运动视频，同时保持上下文相关性、人物外观一致性和背景连续性，实现生成具有出色质量的人类视频。