Diffusion models have demonstrated compelling generation quality by
optimizing the variational lower bound through a simple denoising score
matching loss. In this paper, we provide theoretical evidence that the
prevailing practice of using a constant loss weight strategy in diffusion
models leads to biased estimation during the training phase. Simply optimizing
the denoising network to predict Gaussian noise with constant weighting may
hinder precise estimations of original images. To address the issue, we propose
an elegant and effective weighting strategy grounded in the theoretically
unbiased principle. Moreover, we conduct a comprehensive and systematic
exploration to dissect the inherent bias problem deriving from constant
weighting loss from the perspectives of its existence, impact and reasons.
These analyses are expected to advance our understanding and demystify the
inner workings of diffusion models. Through empirical evaluation, we
demonstrate that our proposed debiased estimation method significantly enhances
sample quality without the reliance on complex techniques, and exhibits
improved efficiency compared to the baseline method both in training and
sampling processes.

通过理论支持，本文提出一种优雅而有效的加权策略，解决了扩散模型中常数加权策略带来的估计偏差问题。并通过实证评估表明，提出的去偏估计方法在样本质量、训练效率和采样效率方面均显著优于基准方法。

去偏扩散模型的训练

Debias the Training of Diffusion Models

The original design of Graph Convolution Network (GCN) couples feature
transformation and neighborhood aggregation for node representation learning.
Recently, some work shows that coupling is inferior to decoupling, which
supports deep graph propagation better and has become the latest paradigm of
GCN (e.g., APPNP and SGCN). Despite effectiveness, the working mechanisms of
the decoupled GCN are not well understood. In this paper, we explore the
decoupled GCN for semi-supervised node classification from a novel and
fundamental perspective -- label propagation. We conduct thorough theoretical
analyses, proving that the decoupled GCN is essentially the same as the
two-step label propagation: first, propagating the known labels along the graph
to generate pseudo-labels for the unlabeled nodes, and second, training normal
neural network classifiers on the augmented pseudo-labeled data. More
interestingly, we reveal the effectiveness of decoupled GCN: going beyond the
conventional label propagation, it could automatically assign structure- and
model- aware weights to the pseudo-label data. This explains why the decoupled
GCN is relatively robust to the structure noise and over-smoothing, but
sensitive to the label noise and model initialization. Based on this insight,
we propose a new label propagation method named Propagation then Training
Adaptively (PTA), which overcomes the flaws of the decoupled GCN with a dynamic
and adaptive weighting strategy. Our PTA is simple yet more effective and
robust than decoupled GCN. We empirically validate our findings on four
benchmark datasets, demonstrating the advantages of our method. The code is
available at this https URL

本文从标签传播的角度探索了解耦后的图卷积网络，证明其本质上与两步标签传播是一致的，并揭示了其有效性，提出了一种新的标签传播方法，Propagation then Training Adaptively (PTA)，通过动态自适应加权策略克服了解耦后的 GCN 的缺陷。该方法在四个基准数据集上得到经验证明优于现有方法。