The vision community has started to build with the recently developed state
space model, Mamba, as the new backbone for a range of tasks. This paper shows
that Mamba's visual capability can be significantly enhanced through
autoregressive pretraining, a direction not previously explored.
Efficiency-wise, the autoregressive nature can well capitalize on the Mamba's
unidirectional recurrent structure, enabling faster overall training speed
compared to other training strategies like mask modeling. Performance-wise,
autoregressive pretraining equips the Mamba architecture with markedly higher
accuracy over its supervised-trained counterparts and, more importantly,
successfully unlocks its scaling potential to large and even huge model sizes.
For example, with autoregressive pretraining, a base-size Mamba attains 83.2\%
ImageNet accuracy, outperforming its supervised counterpart by 2.0\%; our
huge-size Mamba, the largest Vision Mamba to date, attains 85.0\% ImageNet
accuracy (85.5\% when finetuned with $384\times384$ inputs), notably surpassing
all other Mamba variants in vision. The code is available at
https://github.com/OliverRensu/ARM.

Mamba 的视觉能力可以通过自回归预训练显著增强，而且可以成功解锁其大规模模型尺寸的潜力。

用 Mamba 在视觉中进行自回归预训练

Autoregressive Pretraining with Mamba in Vision

This paper enhances image-GPT (iGPT), one of the pioneering works that
introduce autoregressive pretraining to predict next pixels for visual
representation learning. Two simple yet essential changes are made. First, we
shift the prediction target from raw pixels to semantic tokens, enabling a
higher-level understanding of visual content. Second, we supplement the
autoregressive modeling by instructing the model to predict not only the next
tokens but also the visible tokens. This pipeline is particularly effective
when semantic tokens are encoded by discriminatively trained models, such as
CLIP. We introduce this novel approach as D-iGPT. Extensive experiments
showcase that D-iGPT excels as a strong learner of visual representations: A
notable achievement of D-iGPT is its compelling performance on the ImageNet-1K
dataset -- by training on publicly available datasets, D-iGPT achieves 89.5\%
top-1 accuracy with a vanilla ViT-Large model. This model also shows strong
generalization on the downstream task and robustness on out-of-distribution
samples. Code is avaiable at
\href{https://github.com/OliverRensu/D-iGPT}{this https URL}.

本文介绍了 D-iGPT，这是对 image-GPT 进行改进的一种方法，通过将预测目标从原始像素转移到语义标记上，实现对视觉内容的更高级理解，并通过预测可见标记来补充自回归建模，实现了更好的效果。实验证明，D-iGPT 在 ImageNet-1K 数据集上取得了令人瞩目的成绩，并在下游任务中表现出较强的泛化能力和在分布外样本上的鲁棒性。

强大的视觉表达学习者：重塑 Image-GPT

Rejuvenating image-GPT as Strong Visual Representation Learners

With the capability of modeling bidirectional contexts, denoising
autoencoding based pretraining like BERT achieves better performance than
pretraining approaches based on autoregressive language modeling. However,
relying on corrupting the input with masks, BERT neglects dependency between
the masked positions and suffers from a pretrain-finetune discrepancy. In light
of these pros and cons, we propose XLNet, a generalized autoregressive
pretraining method that (1) enables learning bidirectional contexts by
maximizing the expected likelihood over all permutations of the factorization
order and (2) overcomes the limitations of BERT thanks to its autoregressive
formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the
state-of-the-art autoregressive model, into pretraining. Empirically, under
comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a
large margin, including question answering, natural language inference,
sentiment analysis, and document ranking.

我们提出了 XLNet—— 一种广义的自回归预训练方法，它通过最大化分解顺序的所有排列的期望似然来学习双向上下文，并且集成了 Transformer-XL 的思想，从而在 20 项任务中胜过了 BERT，包括问题回答、自然语言推断、情感分析和文档排名等方面。