Segment Anything Model (SAM) has shown impressive zero-shot transfer
performance for various computer vision tasks recently. However, its heavy
computation costs remain daunting for practical applications. MobileSAM
proposes to replace the heavyweight image encoder in SAM with TinyViT by
employing distillation, which results in a significant reduction in
computational requirements. However, its deployment on resource-constrained
mobile devices still encounters challenges due to the substantial memory and
computational overhead caused by self-attention mechanisms. Recently, RepViT
achieves the state-of-the-art performance and latency trade-off on mobile
devices by incorporating efficient architectural designs of ViTs into CNNs.
Here, to achieve real-time segmenting anything on mobile devices, following
MobileSAM, we replace the heavyweight image encoder in SAM with RepViT model,
ending up with the RepViT-SAM model. Extensive experiments show that RepViT-SAM
can enjoy significantly better zero-shot transfer capability than MobileSAM,
along with nearly $10\times$ faster inference speed. The code and models are
available at https://github.com/THU-MIG/RepViT.

为了在资源受限的移动设备上实现实时分割任何物体，本文使用了 RepViT-SAM 模型，代替了 SAM 中的重型图像编码器，得到了具有显著改进的零样本迁移能力和近 10 倍更快推理速度。

RepViT-SAM：面向实时分割任意物体

RepViT-SAM: Towards Real-Time Segmenting Anything

Vision transformer (ViT) recently has drawn great attention in computer
vision due to its remarkable model capability. However, most prevailing ViT
models suffer from huge number of parameters, restricting their applicability
on devices with limited resources. To alleviate this issue, we propose TinyViT,
a new family of tiny and efficient small vision transformers pretrained on
large-scale datasets with our proposed fast distillation framework. The central
idea is to transfer knowledge from large pretrained models to small ones, while
enabling small models to get the dividends of massive pretraining data. More
specifically, we apply distillation during pretraining for knowledge transfer.
The logits of large teacher models are sparsified and stored in disk in advance
to save the memory cost and computation overheads. The tiny student
transformers are automatically scaled down from a large pretrained model with
computation and parameter constraints. Comprehensive experiments demonstrate
the efficacy of TinyViT. It achieves a top-1 accuracy of 84.8% on ImageNet-1k
with only 21M parameters, being comparable to Swin-B pretrained on ImageNet-21k
while using 4.2 times fewer parameters. Moreover, increasing image resolutions,
TinyViT can reach 86.5% accuracy, being slightly better than Swin-L while using
only 11% parameters. Last but not the least, we demonstrate a good transfer
ability of TinyViT on various downstream tasks. Code and models are available
at this https URL

TinyViT 是一种新型的、基于快速蒸馏框架的小型视觉 Transformer，通过使用蒸馏预训练模型的知识迁移以及对计算和参数进行限制，能够在减少参数数量的情况下依然具有较高的准确性，并且能够在多种下游任务中得到很好的转移效果。