The development of autoregressive modeling (AM) in computer vision lags
behind natural language processing (NLP) in self-supervised pre-training. This
is mainly caused by the challenge that images are not sequential signals and
lack a natural order when applying autoregressive modeling. In this study,
inspired by human beings' way of grasping an image, i.e., focusing on the main
object first, we present a semantic-aware autoregressive image modeling
(SemAIM) method to tackle this challenge. The key insight of SemAIM is to
autoregressive model images from the semantic patches to the less semantic
patches. To this end, we first calculate a semantic-aware permutation of
patches according to their feature similarities and then perform the
autoregression procedure based on the permutation. In addition, considering
that the raw pixels of patches are low-level signals and are not ideal
prediction targets for learning high-level semantic representation, we also
explore utilizing the patch features as the prediction targets. Extensive
experiments are conducted on a broad range of downstream tasks, including image
classification, object detection, and instance/semantic segmentation, to
evaluate the performance of SemAIM. The results demonstrate SemAIM achieves
state-of-the-art performance compared with other self-supervised methods.
Specifically, with ViT-B, SemAIM achieves 84.1% top-1 accuracy for fine-tuning
on ImageNet, 51.3% AP and 45.4% AP for object detection and instance
segmentation on COCO, which outperforms the vanilla MAE by 0.5%, 1.0%, and
0.5%, respectively.

自主训练自回归模型在计算机视觉中的发展滞后于自然语言处理，本研究提出了一种基于语义感知的自回归图像建模方法，通过对补丁的特征相似性进行排序，将自回归建模从语义补丁扩展到非语义补丁，并探索了使用补丁特征作为预测目标的方法，实验表明该方法在图像分类、目标检测和分割等多个任务中实现了卓越的性能表现。