At the core of both successful generative and self-supervised representation
learning models there is a reconstruction objective that incorporates some form
of image corruption. Diffusion models implement this approach through a
scheduled Gaussian corruption process, while masked auto-encoder models do so
by masking patches of the image. Despite their different approaches, the
underlying similarity in their methodologies suggests a promising avenue for an
auto-encoder capable of both de-noising tasks. We propose a unified
self-supervised objective, dubbed Unified Masked Diffusion (UMD), that combines
patch-based and noise-based corruption techniques within a single auto-encoding
framework. Specifically, UMD modifies the diffusion transformer (DiT) training
process by introducing an additional noise-free, high masking representation
step in the diffusion noising schedule, and utilizes a mixed masked and noised
image for subsequent timesteps. By integrating features useful for diffusion
modeling and for predicting masked patch tokens, UMD achieves strong
performance in downstream generative and representation learning tasks,
including linear probing and class-conditional generation. This is achieved
without the need for heavy data augmentations, multiple views, or additional
encoders. Furthermore, UMD improves over the computational efficiency of prior
diffusion based methods in total training time. We release our code at
this https URL

提出了一种统一的自监督目标（Unified Masked Diffusion），通过在单一的自编码框架中结合基于补丁和噪声的破坏技术，使用扩散变换器（DiT）培训过程中引入额外的无噪声、高掩膜表示步骤，并在后续时间步骤中使用混合的掩膜和噪声图像，以实现在下游生成和表示学习任务中强大的性能。