Autoregressive language modeling (ALM) have been successfully used in
self-supervised pre-training in Natural language processing (NLP). However,
this paradigm has not achieved comparable results with other self-supervised
approach in computer vision (e.g., contrastive learning, mask image modeling).
In this paper, we try to find the reason why autoregressive modeling does not
work well on vision tasks. To tackle this problem, we fully analyze the
limitation of visual autoregressive methods and proposed a novel stochastic
autoregressive image modeling (named SAIM) by the two simple designs. First, we
employ stochastic permutation strategy to generate effective and robust image
context which is critical for vision tasks. Second, we create a parallel
encoder-decoder training process in which the encoder serves a similar role to
the standard vision transformer focus on learning the whole contextual
information, and meanwhile the decoder predicts the content of the current
position, so that the encoder and decoder can reinforce each other. By
introducing stochastic prediction and the parallel encoder-decoder, SAIM
significantly improve the performance of autoregressive image modeling. Our
method achieves the best accuracy (83.9%) on the vanilla ViT-Base model among
methods using only ImageNet-1K data. Transfer performance in downstream tasks
also show that our model achieves competitive performance.

本文针对自回归模型在计算机视觉领域无法与其他自监督学习方法相比的问题，通过引入随机排列策略和并行编码 - 解码训练过程，提出了一种名为 SAIM 的新型随机自回归图像建模方法，取得了极大的性能提升，特别是在基于 ImageNet-1K 数据的视觉变换任务方面达到最佳精度 83.9%。