Image recognition and generation have long been developed independently of
each other. With the recent trend towards general-purpose representation
learning, the development of general representations for both recognition and
generation tasks is also promoted. However, preliminary attempts mainly focus
on generation performance, but are still inferior on recognition tasks. These
methods are modeled in the vector-quantized (VQ) space, whereas leading
recognition methods use pixels as inputs. Our key insights are twofold: (1)
pixels as inputs are crucial for recognition tasks; (2) VQ tokens as
reconstruction targets are beneficial for generation tasks. These observations
motivate us to propose an Alternating Denoising Diffusion Process (ADDP) that
integrates these two spaces within a single representation learning framework.
In each denoising step, our method first decodes pixels from previous VQ
tokens, then generates new VQ tokens from the decoded pixels. The diffusion
process gradually masks out a portion of VQ tokens to construct the training
samples. The learned representations can be used to generate diverse
high-fidelity images and also demonstrate excellent transfer performance on
recognition tasks. Extensive experiments show that our method achieves
competitive performance on unconditional generation, ImageNet classification,
COCO detection, and ADE20k segmentation. Importantly, our method represents the
first successful development of general representations applicable to both
generation and dense recognition tasks. Code shall be released.

本文提出了一种交替去噪扩散过程 (ADDP)，该过程将像素和矢量量化 (VQ) 空间相结合，用于图像识别和生成任务中的表示学习，并取得了在各种任务中的成功结果。