The strength of modern large-scale neural networks lies in their ability to
efficiently adapt to new tasks with few examples. Although extensive research
has investigated the transferability of Vision Transformers (ViTs) to various
downstream tasks under diverse constraints, this study shifts focus to explore
the transfer learning potential of [V]-Mamba. We compare its performance with
ViTs across different few-shot data budgets and efficient transfer methods. Our
analysis yields three key insights into [V]-Mamba's few-shot transfer
performance: (a) [V]-Mamba demonstrates superior or equivalent few-shot
learning capabilities compared to ViTs when utilizing linear probing (LP) for
transfer, (b) Conversely, [V]-Mamba exhibits weaker or similar few-shot
learning performance compared to ViTs when employing visual prompting (VP) as
the transfer method, and (c) We observe a weak positive correlation between the
performance gap in transfer via LP and VP and the scale of the [V]-Mamba model.
This preliminary analysis lays the foundation for more comprehensive studies
aimed at furthering our understanding of the capabilities of [V]-Mamba variants
and their distinctions from ViTs.

本文研究了大规模神经网络在少样本迁移学习中的能力，通过对 [V]-Mamba 模型与 ViTs 模型在不同少样本数据预算和高效迁移方法下的性能比较，发现 [V]-Mamba 在线性探测（LP）迁移方案下表现优于或等同于 ViTs，而在视觉提示（VP）迁移方法下表现较弱或类似于 ViTs，同时 [V]-Mamba 模型的规模与迁移方案之间存在弱正相关性。这项初步分析为进一步研究 [V]-Mamba 变体和 ViTs 之间的差异，以及它们的能力提供了基础。

关于 [V]-Mamba 的低样本迁移能力

On the low-shot transferability of [V]-Mamba

Scaling up deep neural network capacity has been known as an effective
approach to improving model quality for several different machine learning
tasks. In many cases, increasing model capacity beyond the memory limit of a
single accelerator has required developing special algorithms or
infrastructure. These solutions are often architecture-specific and do not
transfer to other tasks. To address the need for efficient and task-independent
model parallelism, we introduce GPipe, a pipeline parallelism library that
allows scaling any network that can be expressed as a sequence of layers. By
pipelining different sub-sequences of layers on separate accelerators, GPipe
provides the flexibility of scaling a variety of different networks to gigantic
sizes efficiently. Moreover, GPipe utilizes a novel batch-splitting pipelining
algorithm, resulting in almost linear speedup when a model is partitioned
across multiple accelerators. We demonstrate the advantages of GPipe by
training large-scale neural networks on two different tasks with distinct
network architectures: (i) Image Classification: We train a
557-million-parameter AmoebaNet model and attain a top-1 accuracy of 84.4% on
ImageNet-2012, (ii) Multilingual Neural Machine Translation: We train a single
6-billion-parameter, 128-layer Transformer model on a corpus spanning over 100
languages and achieve better quality than all bilingual models.

GPipe 是一种管道并行库，它提供了一种高效进行模型并行化的方式，可以将不同子层序列分别放在不同的加速器上，因此可以有效地将各种不同的网络扩展到巨大的规模，并且能够实现快速的线性加速，通过对两个不同任务的训练，证明了 GPipe 的优势。