State space models and Mamba-based models have been increasingly applied
across various domains, achieving state-of-the-art performance. This technical
report introduces the first attempt to train a transferable Mamba model
utilizing contrastive language-image pretraining (CLIP). We have trained Mamba
models of varying sizes and undertaken comprehensive evaluations of these
models on 26 zero-shot classification datasets and 16 out-of-distribution (OOD)
datasets. Our findings reveal that a Mamba model with 67 million parameters is
on par with a 307 million-parameter Vision Transformer (ViT) model in zero-shot
classification tasks, highlighting the parameter efficiency of Mamba models. In
tests of OOD generalization, Mamba-based models exhibit exceptional performance
in conditions of OOD image contrast or when subjected to high-pass filtering.
However, a Hessian analysis indicates that Mamba models feature a sharper and
more non-convex landscape compared to ViT-based models, making them more
challenging to train. The source code is available at
this https URL

通过对比度语言图像预训练技术，本技术报告首次引入了训练可传输 Mamba 模型的方法。在 26 个零样本分类数据集和 16 个超出分布范围的数据集上，我们训练了不同规模的 Mamba 模型，并对其进行了全面评估。结果表明，具有 6700 万参数的 Mamba 模型在零样本分类任务中与具有 3.07 亿参数的 Vision Transformer 模型相媲美，突显了 Mamba 模型的参数效率。在超出分布范围的泛化测试中，基于 Mamba 的模型在图像对比度差异或经过高通滤波时表现出色。然而，Hessian 分析表明，与 ViT 模型相比，Mamba 模型的景观更加尖锐和非凸，使其更具挑战性。源代码可从此 URL 获得。