This paper enhances image-GPT (iGPT), one of the pioneering works that
introduce autoregressive pretraining to predict next pixels for visual
representation learning. Two simple yet essential changes are made. First, we
shift the prediction target from raw pixels to semantic tokens, enabling a
higher-level understanding of visual content. Second, we supplement the
autoregressive modeling by instructing the model to predict not only the next
tokens but also the visible tokens. This pipeline is particularly effective
when semantic tokens are encoded by discriminatively trained models, such as
CLIP. We introduce this novel approach as D-iGPT. Extensive experiments
showcase that D-iGPT excels as a strong learner of visual representations: A
notable achievement of D-iGPT is its compelling performance on the ImageNet-1K
dataset -- by training on publicly available datasets, D-iGPT achieves 89.5\%
top-1 accuracy with a vanilla ViT-Large model. This model also shows strong
generalization on the downstream task and robustness on out-of-distribution
samples. Code is avaiable at
\href{https://github.com/OliverRensu/D-iGPT}{this https URL}.

本文介绍了 D-iGPT，这是对 image-GPT 进行改进的一种方法，通过将预测目标从原始像素转移到语义标记上，实现对视觉内容的更高级理解，并通过预测可见标记来补充自回归建模，实现了更好的效果。实验证明，D-iGPT 在 ImageNet-1K 数据集上取得了令人瞩目的成绩，并在下游任务中表现出较强的泛化能力和在分布外样本上的鲁棒性。