In this work, we propose a novel Trajectory Score Matching (TSM) method that
aims to solve the pseudo ground truth inconsistency problem caused by the
accumulated error in Interval Score Matching (ISM) when using the Denoising
Diffusion Implicit Models (DDIM) inversion process. Unlike ISM which adopts the
inversion process of DDIM to calculate on a single path, our TSM method
leverages the inversion process of DDIM to generate two paths from the same
starting point for calculation. Since both paths start from the same starting
point, TSM can reduce the accumulated error compared to ISM, thus alleviating
the problem of pseudo ground truth inconsistency. TSM enhances the stability
and consistency of the model's generated paths during the distillation process.
We demonstrate this experimentally and further show that ISM is a special case
of TSM. Furthermore, to optimize the current multi-stage optimization process
from high-resolution text to 3D generation, we adopt Stable Diffusion XL for
guidance. In response to the issues of abnormal replication and splitting
caused by unstable gradients during the 3D Gaussian splatting process when
using Stable Diffusion XL, we propose a pixel-by-pixel gradient clipping
method. Extensive experiments show that our model significantly surpasses the
state-of-the-art models in terms of visual quality and performance. Code:
https://github.com/xingy038/Dreamer-XL.

该研究提出了一种新颖的轨迹分数匹配方法 (Trajectory Score Matching, TSM)，旨在解决使用去噪扩散隐式模型 (Denoising Diffusion Implicit Models, DDIM) 反演过程中，Interval Score Matching (ISM) 由于累积误差导致的伪真值不一致问题。通过利用 DDIM 的反演过程计算相同起点的两条路径，TSM 可以减小与 ISM 相比的累积误差，从而缓解伪真值不一致问题。TSM 提高了模型在蒸馏过程中生成路径的稳定性和一致性，并实验证明 ISM 是 TSM 的特例。此外，为了优化从高分辨率文本到三维生成的多阶段优化过程，我们采用了稳定扩散 XL 进行指导。针对使用稳定扩散 XL 时由于不稳定梯度导致的异常复制和分裂问题，我们提出了一种逐像素梯度裁剪方法。广泛的实验证明我们的模型在视觉质量和性能方面显著超过了目前的最先进模型。

Dreamer XL：通过轨迹得分匹配实现高分辨率文本到 3D 生成

Dreamer XL: Towards High-Resolution Text-to-3D Generation via Trajectory  Score Matching

Stable Diffusion XL (SDXL) has become the best open source text-to-image
model (T2I) for its versatility and top-notch image quality. Efficiently
addressing the computational demands of SDXL models is crucial for wider reach
and applicability. In this work, we introduce two scaled-down variants, Segmind
Stable Diffusion (SSD-1B) and Segmind-Vega, with 1.3B and 0.74B parameter
UNets, respectively, achieved through progressive removal using layer-level
losses focusing on reducing the model size while preserving generative quality.
We release these models weights at this https URL Our methodology
involves the elimination of residual networks and transformer blocks from the
U-Net structure of SDXL, resulting in significant reductions in parameters, and
latency. Our compact models effectively emulate the original SDXL by
capitalizing on transferred knowledge, achieving competitive results against
larger multi-billion parameter SDXL. Our work underscores the efficacy of
knowledge distillation coupled with layer-level losses in reducing model size
while preserving the high-quality generative capabilities of SDXL, thus
facilitating more accessible deployment in resource-constrained environments.

通过降低模型大小和知识蒸馏，我们引入了两种简化的 Stable Diffusion XL 模型 (SSD-1B 和 Segmind-Vega)，并证明了在保持高质量生成能力的同时减少模型大小的有效性。

稳定扩散 XL 的渐进式知识蒸馏以层级损失

Progressive Knowledge Distillation Of Stable Diffusion XL Using Layer  Level Loss

Stable diffusion is the mainstay of the text-to-image (T2I) synthesis in the
community due to its generation performance and open-source nature. Recently,
Stable Diffusion XL (SDXL), the successor of stable diffusion, has received a
lot of attention due to its significant performance improvements with a higher
resolution of 1024x1024 and a larger model. However, its increased computation
cost and model size require higher-end hardware(e.g., bigger VRAM GPU) for
end-users, incurring higher costs of operation. To address this problem, in
this work, we propose an efficient latent diffusion model for text-to-image
synthesis obtained by distilling the knowledge of SDXL. To this end, we first
perform an in-depth analysis of the denoising U-Net in SDXL, which is the main
bottleneck of the model, and then design a more efficient U-Net based on the
analysis. Secondly, we explore how to effectively distill the generation
capability of SDXL into an efficient U-Net and eventually identify four
essential factors, the core of which is that self-attention is the most
important part. With our efficient U-Net and self-attention-based knowledge
distillation strategy, we build our efficient T2I models, called KOALA-1B &
-700M, while reducing the model size up to 54% and 69% of the original SDXL
model. In particular, the KOALA-700M is more than twice as fast as SDXL while
still retaining a decent generation quality. We hope that due to its balanced
speed-performance tradeoff, our KOALA models can serve as a cost-effective
alternative to SDXL in resource-constrained environments.

通过蒸馏 SDXL 模型的知识，我们提出了一种高效的文本到图像综合的潜在扩散模型，其核心是通过设计有效的 U-Net 和自注意力的知识蒸馏策略，使 KOALA 模型在减小模型尺寸的同时，仍保持良好的生成质量，相较于 SDXL 模型有两倍以上的速度，可以在资源有限的环境中成为一种具有成本效益的替代选择。