Diffusion Transformer (DiT) has emerged as the new trend of generative
diffusion models on image generation. In view of extremely slow convergence in
typical DiT, recent breakthroughs have been driven by mask strategy that
significantly improves the training efficiency of DiT with additional
intra-image contextual learning. Despite this progress, mask strategy still
suffers from two inherent limitations: (a) training-inference discrepancy and
(b) fuzzy relations between mask reconstruction & generative diffusion process,
resulting in sub-optimal training of DiT. In this work, we address these
limitations by novelly unleashing the self-supervised discrimination knowledge
to boost DiT training. Technically, we frame our DiT in a teacher-student
manner. The teacher-student discriminative pairs are built on the diffusion
noises along the same Probability Flow Ordinary Differential Equation (PF-ODE).
Instead of applying mask reconstruction loss over both DiT encoder and decoder,
we decouple DiT encoder and decoder to separately tackle discriminative and
generative objectives. In particular, by encoding discriminative pairs with
student and teacher DiT encoders, a new discriminative loss is designed to
encourage the inter-image alignment in the self-supervised embedding space.
After that, student samples are fed into student DiT decoder to perform the
typical generative diffusion task. Extensive experiments are conducted on
ImageNet dataset, and our method achieves a competitive balance between
training cost and generative capacity.

通过引入自监督区分知识来增强扩散变压器 (Diffusion Transformer, DiT) 的训练效果，并在 ImageNet 数据集上进行了广泛实验，实现了训练成本和生成能力之间的竞争平衡。