Segment Anything Model (SAM) has achieved impressive performance in many
computer vision tasks. However, as a large-scale model, the immense memory and
computation costs hinder its practical deployment. In this paper, we propose a
post-training quantization (PTQ) framework for Segment Anything Model, namely
PTQ4SAM. First, we investigate the inherent bottleneck of SAM quantization
attributed to the bimodal distribution in post-Key-Linear activations. We
analyze its characteristics from both per-tensor and per-channel perspectives,
and propose a Bimodal Integration strategy, which utilizes a mathematically
equivalent sign operation to transform the bimodal distribution into a
relatively easy-quantized normal distribution offline. Second, SAM encompasses
diverse attention mechanisms (i.e., self-attention and two-way
cross-attention), resulting in substantial variations in the post-Softmax
distributions. Therefore, we introduce an Adaptive Granularity Quantization for
Softmax through searching the optimal power-of-two base, which is
hardware-friendly. Extensive experimental results across various vision tasks
(instance segmentation, semantic segmentation and object detection), datasets
and model variants show the superiority of PTQ4SAM. For example, when
quantizing SAM-L to 6-bit, we achieve lossless accuracy for instance
segmentation, about 0.5\% drop with theoretical 3.9$\times$ acceleration. The
code is available at https://github.com/chengtao-lv/PTQ4SAM.

在这篇论文中，我们提出了一种针对 Segment Anything Model 的后训练量化框架，即 PTQ4SAM。我们通过分析 SAM 量化中的双峰分布特性，提出了双峰积分策略，并采用适应性颗粒度量化方法来处理 SAM 中的后 Softmax 分布，实验证明 PTQ4SAM 在各种视觉任务和模型变体中具有卓越的优势。

PTQ4SAM：用于分段任意物体的训练后量化

PTQ4SAM: Post-Training Quantization for Segment Anything

Post-training quantization (PTQ) reduces the memory footprint of LLMs by
quantizing their weights to low-precision. In this work, we introduce QuIP#, a
weight-only PTQ method that achieves state-of-the-art results in extreme
compression regimes ($\le$ 4 bits per weight) using three novel techniques.
First, QuIP# improves the incoherence processing from QuIP by using the
randomized Hadamard transform, which is faster and has better theoretical
properties. Second, QuIP# uses vector quantization techniques to take advantage
of the ball-shaped sub-Gaussian distribution that incoherent weights possess:
specifically, we introduce a set of hardware-efficient codebooks based on the
highly symmetric $E_8$ lattice, which achieves the optimal 8-dimension unit
ball packing. Third, QuIP# uses fine-tuning to improve fidelity to the original
model. Our experiments show that QuIP# outperforms existing PTQ methods,
enables new behaviors in PTQ scaling, and supports fast inference.

在这项研究中，我们引入了权重量化的新方法（QuIP#），通过三个创新技术，在极端压缩模式下（每个权重≤4 位）实现了最先进的结果。QuIP# 通过使用随机哈达玛变换改进 QuIP 的不相干处理，并利用具有不相干权重的球形亚高斯分布来使用向量量化技术。此外，QuIP# 使用微调来提高对原始模型的保真度，实验结果显示其超过了现有的 PTQ 方法，支持快速推理。

QuIP#: 通过哈达玛不相干和格码书更好地 LLM 量化

QuIP#: Even Better LLM Quantization with Hadamard Incoherence and  Lattice Codebooks

We study the challenging task of neural network quantization without
end-to-end retraining, called Post-training Quantization (PTQ). PTQ usually
requires a small subset of training data but produces less powerful quantized
models than Quantization-Aware Training (QAT). In this work, we propose a novel
PTQ framework, dubbed BRECQ, which pushes the limits of bitwidth in PTQ down to
INT2 for the first time. BRECQ leverages the basic building blocks in neural
networks and reconstructs them one-by-one. In a comprehensive theoretical study
of the second-order error, we show that BRECQ achieves a good balance between
cross-layer dependency and generalization error. To further employ the power of
quantization, the mixed precision technique is incorporated in our framework by
approximating the inter-layer and intra-layer sensitivity. Extensive
experiments on various handcrafted and searched neural architectures are
conducted for both image classification and object detection tasks. And for the
first time we prove that, without bells and whistles, PTQ can attain 4-bit
ResNet and MobileNetV2 comparable with QAT and enjoy 240 times faster
production of quantized models. Codes are available at
this https URL

本文介绍了一种名为 BRECQ 的新型 PTQ 框架，通过重构神经网络的组件来将 PTQ 的位宽降至 INT2，并将混合精度技术结合到框架中。对多种手工制作和搜索的神经体系结构进行了广泛的实验，证明了 PTQ 可以实现比使用 QAT 的 4 位 ResNet 和 MobileNetV2 更快的速度制作量化模型。