Segment Anything Model (SAM) has achieved impressive performance in many
computer vision tasks. However, as a large-scale model, the immense memory and
computation costs hinder its practical deployment. In this paper, we propose a
post-training quantization (PTQ) framework for Segment Anything Model, namely
PTQ4SAM. First, we investigate the inherent bottleneck of SAM quantization
attributed to the bimodal distribution in post-Key-Linear activations. We
analyze its characteristics from both per-tensor and per-channel perspectives,
and propose a Bimodal Integration strategy, which utilizes a mathematically
equivalent sign operation to transform the bimodal distribution into a
relatively easy-quantized normal distribution offline. Second, SAM encompasses
diverse attention mechanisms (i.e., self-attention and two-way
cross-attention), resulting in substantial variations in the post-Softmax
distributions. Therefore, we introduce an Adaptive Granularity Quantization for
Softmax through searching the optimal power-of-two base, which is
hardware-friendly. Extensive experimental results across various vision tasks
(instance segmentation, semantic segmentation and object detection), datasets
and model variants show the superiority of PTQ4SAM. For example, when
quantizing SAM-L to 6-bit, we achieve lossless accuracy for instance
segmentation, about 0.5\% drop with theoretical 3.9$\times$ acceleration. The
code is available at https://github.com/chengtao-lv/PTQ4SAM.

在这篇论文中，我们提出了一种针对 Segment Anything Model 的后训练量化框架，即 PTQ4SAM。我们通过分析 SAM 量化中的双峰分布特性，提出了双峰积分策略，并采用适应性颗粒度量化方法来处理 SAM 中的后 Softmax 分布，实验证明 PTQ4SAM 在各种视觉任务和模型变体中具有卓越的优势。