Segment Anything Model (SAM) has emerged as a powerful tool for numerous
vision applications. A key component that drives the impressive performance for
zero-shot transfer and high versatility is a super large Transformer model
trained on the extensive high-quality SA-1B dataset. While beneficial, the huge
computation cost of SAM model has limited its applications to wider real-world
applications. To address this limitation, we propose EfficientSAMs,
light-weight SAM models that exhibits decent performance with largely reduced
complexity. Our idea is based on leveraging masked image pretraining, SAMI,
which learns to reconstruct features from SAM image encoder for effective
visual representation learning. Further, we take SAMI-pretrained light-weight
image encoders and mask decoder to build EfficientSAMs, and finetune the models
on SA-1B for segment anything task. We perform evaluations on multiple vision
tasks including image classification, object detection, instance segmentation,
and semantic object detection, and find that our proposed pretraining method,
SAMI, consistently outperforms other masked image pretraining methods. On
segment anything task such as zero-shot instance segmentation, our
EfficientSAMs with SAMI-pretrained lightweight image encoders perform favorably
with a significant gain (e.g., ~4 AP on COCO/LVIS) over other fast SAM models.

基于大规模 Transformer 模型的 SAM 模型制约了其在广泛现实应用中的计算成本。为解决这个问题，我们提出了轻量级 SAM 模型 EfficientSAMs，借助图像预训练方法 SAMI，通过 SAM 图像编码器重建特征进行视觉表征学习，并在 SA-1B 上微调模型，实现对图像分类、对象检测、实例分割和语义对象检测等多个视觉任务的评估。结果显示，SAMI 方法在面向零样本实例分割等任务上，我们的 EfficientSAMs 表现优于其他快速 SAM 模型，达到显著增益（例如，在 COCO/LVIS 上的 AP 提高了约 4 个点）。

高效 SAM：基于掩码图像预训练的高效物体分割

EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment  Anything

Most invariance-based self-supervised methods rely on single object-centric
images (e.g., ImageNet images) for pretraining, learning invariant
representations from geometric transformations. However, when images are not
object-centric, the semantics of the image can be significantly altered due to
cropping. Furthermore, as the model learns geometrically insensitive features,
it may struggle to capture location information. For this reason, we propose a
Geometric Transformation Sensitive Architecture that learns features sensitive
to geometric transformations, specifically four-fold rotation, random crop, and
multi-crop. Our method encourages the student to learn sensitive features by
using targets that are sensitive to those transforms via pooling and rotating
of the teacher feature map and predicting rotation. Additionally, since
training insensitively to multi-crop can capture long-term dependencies, we use
patch correspondence loss to train the model sensitively while capturing
long-term dependencies. Our approach demonstrates improved performance when
using non-object-centric images as pretraining data compared to other methods
that learn geometric transformation-insensitive representations. We surpass the
DINO[\citet{caron2021emerging}] baseline in tasks including image
classification, semantic segmentation, detection, and instance segmentation
with improvements of 6.1 $Acc$, 3.3 $mIoU$, 3.4 $AP^b$, and 2.7 $AP^m$. Code
and pretrained models are publicly available at:

提出了一种几何变换敏感的模型结构，用于自我监督图像预训练，可在非物体中心图像上提供更好的性能表现，可应用于图像分类、语义分割、检测和实例分割等多项任务。