The recently proposed SparseFormer architecture provides an alternative
approach to visual understanding by utilizing a significantly lower number of
visual tokens via adjusting RoIs, greatly reducing computational costs while
still achieving promising performance. However, training SparseFormers from
scratch is still expensive, and scaling up the number of parameters can be
challenging. In this paper, we propose to bootstrap SparseFormers from
ViT-based vision foundation models in a simple and efficient way. Since the
majority of SparseFormer blocks are the standard transformer ones, we can
inherit weights from large-scale pre-trained vision transformers and freeze
them as much as possible. Therefore, we only need to train the
SparseFormer-specific lightweight focusing transformer to adjust token RoIs and
fine-tune a few early pre-trained blocks to align the final token
representation. In such a way, we can bootstrap SparseFormer architectures from
various large-scale pre-trained models (e.g., IN-21K pre-trained AugRegs or
CLIPs) using a rather smaller amount of training samples (e.g., IN-1K) and
without labels or captions within just a few hours. As a result, the
bootstrapped unimodal SparseFormer (from AugReg-ViT-L/16-384) can reach 84.9%
accuracy on IN-1K with only 49 tokens, and the multimodal SparseFormer from
CLIPs also demonstrates notable zero-shot performance with highly reduced
computational cost without seeing any caption during the bootstrapping
procedure. In addition, CLIP-bootstrapped SparseFormers, which align the output
space with language without seeing a word, can serve as efficient vision
encoders in multimodal large language models. Code will be publicly available
at this https URL

用于视觉理解的 SparseFormer 通过调整 RoIs 利用较少的视觉标记提供了一种替代方案，从而大大减少了计算成本并同时实现了有希望的性能。我们提出了一种简单高效的方法，通过从基于 ViT 的视觉基础模型中启动 SparseFormer，仅需训练 SparseFormer 特定的轻量级聚焦变换器，从而可以以较小数量的训练样本且不需要标签或标题，在几小时内构建具有不同大规模预训练模型的 SparseFormer 架构。这种方法可以在 IN-1K 上使用仅 49 个标记达到 84.9％的准确率，并且基于 CLIP 的多模态 SparseFormer 还能在具有大大降低的计算成本的情况下展示出显著的零样本性能，而尚未在启动过程中查看任何标题。此外，基于 CLIP 启动的 SparseFormers 可以将输出空间与语言对齐，成为多模态大型语言模型中高效的视觉编码器。

从视觉基础模型起步的稀疏 Transformer

Bootstrapping SparseFormers from Vision Foundation Models

Human visual recognition is a sparse process, where only a few salient visual
cues are attended to rather than traversing every detail uniformly. However,
most current vision networks follow a dense paradigm, processing every single
visual unit (e.g,, pixel or patch) in a uniform manner. In this paper, we
challenge this dense paradigm and present a new method, coined SparseFormer, to
imitate human's sparse visual recognition in an end-to-end manner. SparseFormer
learns to represent images using a highly limited number of tokens (down to 49)
in the latent space with sparse feature sampling procedure instead of
processing dense units in the original pixel space. Therefore, SparseFormer
circumvents most of dense operations on the image space and has much lower
computational costs. Experiments on the ImageNet classification benchmark
dataset show that SparseFormer achieves performance on par with canonical or
well-established models while offering better accuracy-throughput tradeoff.
Moreover, the design of our network can be easily extended to the video
classification with promising performance at lower computational costs. We hope
that our work can provide an alternative way for visual modeling and inspire
further research on sparse neural architectures. The code will be publicly
available at this https URL

本文提出了一种命名为 SparseFormer 的新方法，以在端到端的方式中模仿人的稀疏视觉识别，其中 SparseFormer 使用稀疏特征抽样过程，在潜在空间中使用极少量的标记（降至 49），而不是在原始像素空间中处理密集单元，从而具有更低的计算成本。SparseFormer 在 ImageNet 分类基准数据集上的实验表明，它具有与规范或已建立模型相当的性能，同时提供更好的准确度 - 吞吐量权衡，并且设计我们的网络可以以更低的计算成本轻松扩展到视频分类。