The vision community has started to build with the recently developed state
space model, Mamba, as the new backbone for a range of tasks. This paper shows
that Mamba's visual capability can be significantly enhanced through
autoregressive pretraining, a direction not previously explored.
Efficiency-wise, the autoregressive nature can well capitalize on the Mamba's
unidirectional recurrent structure, enabling faster overall training speed
compared to other training strategies like mask modeling. Performance-wise,
autoregressive pretraining equips the Mamba architecture with markedly higher
accuracy over its supervised-trained counterparts and, more importantly,
successfully unlocks its scaling potential to large and even huge model sizes.
For example, with autoregressive pretraining, a base-size Mamba attains 83.2\%
ImageNet accuracy, outperforming its supervised counterpart by 2.0\%; our
huge-size Mamba, the largest Vision Mamba to date, attains 85.0\% ImageNet
accuracy (85.5\% when finetuned with $384\times384$ inputs), notably surpassing
all other Mamba variants in vision. The code is available at
https://github.com/OliverRensu/ARM.

Mamba 的视觉能力可以通过自回归预训练显著增强，而且可以成功解锁其大规模模型尺寸的潜力。

用 Mamba 在视觉中进行自回归预训练

Autoregressive Pretraining with Mamba in Vision

Despite the empirical success of neural architecture search (NAS) in deep
learning applications, the optimality, reproducibility and cost of NAS schemes
remain hard to assess. In this paper, we propose Generative Adversarial NAS
(GA-NAS) with theoretically provable convergence guarantees, promoting
stability and reproducibility in neural architecture search. Inspired by
importance sampling, GA-NAS iteratively fits a generator to previously
discovered top architectures, thus increasingly focusing on important parts of
a large search space. Furthermore, we propose an efficient adversarial learning
approach, where the generator is trained by reinforcement learning based on
rewards provided by a discriminator, thus being able to explore the search
space without evaluating a large number of architectures. Extensive experiments
show that GA-NAS beats the best published results under several cases on three
public NAS benchmarks. In the meantime, GA-NAS can handle ad-hoc search
constraints and search spaces. We show that GA-NAS can be used to improve
already optimized baselines found by other NAS methods, including EfficientNet
and ProxylessNAS, in terms of ImageNet accuracy or the number of parameters, in
their original search space.

本文提出了具有可证明收敛保证的生成式对抗网络 NAS (GA-NAS)，并基于权重的鉴别器提供的奖励，采用增强学习训练生成器，探索搜寻空间而无需评估大量体系结构，通过三个公共 NAS 基准测试，在多个情况下 GA-NAS 都能打败已发布的最佳结果，同时 GA-NAS 可以处理特定搜寻约束和空间，能用于改进其他 NAS 方法找到的优化基线的 ImageNet 准确性或参数数量。