Diffusion models, emerging as powerful deep generative tools, excel in
various applications. They operate through a two-steps process: introducing
noise into training samples and then employing a model to convert random noise
into new samples (e.g., images). However, their remarkable generative
performance is hindered by slow training and sampling. This is due to the
necessity of tracking extensive forward and reverse diffusion trajectories, and
employing a large model with numerous parameters across multiple timesteps
(i.e., noise levels). To tackle these challenges, we present a multi-stage
framework inspired by our empirical findings. These observations indicate the
advantages of employing distinct parameters tailored to each timestep while
retaining universal parameters shared across all time steps. Our approach
involves segmenting the time interval into multiple stages where we employ
custom multi-decoder U-net architecture that blends time-dependent models with
a universally shared encoder. Our framework enables the efficient distribution
of computational resources and mitigates inter-stage interference, which
substantially improves training efficiency. Extensive numerical experiments
affirm the effectiveness of our framework, showcasing significant training and
sampling efficiency enhancements on three state-of-the-art diffusion models,
including large-scale latent diffusion models. Furthermore, our ablation
studies illustrate the impact of two important components in our framework: (i)
a novel timestep clustering algorithm for stage division, and (ii) an
innovative multi-decoder U-net architecture, seamlessly integrating universal
and customized hyperparameters.

通过多阶段框架和多解码器 U-net 架构，我们提出了一种增强扩散模型训练和采样效率的方案，通过定制每个时间步长的不同参数，同时保留所有时间步长共享的通用参数，有效地分配计算资源并减轻阶段间干扰。