Compression of large and performant vision foundation models (VFMs) into
arbitrary bit-wise operations (BitOPs) allows their deployment on various
hardware. We propose to fine-tune a VFM to a mixed-precision quantized
supernet. The supernet-based neural architecture search (NAS) can be adopted
for this purpose, which trains a supernet, and then subnets within arbitrary
hardware budgets can be extracted. However, existing methods face difficulties
in optimizing the mixed-precision search space and incurring large memory costs
during training. To tackle these challenges, first, we study the effective
search space design for fine-tuning a VFM by comparing different operators
(such as resolution, feature size, width, depth, and bit-widths) in terms of
performance and BitOPs reduction. Second, we propose memory-efficient supernet
training using a low-rank adapter (LoRA) and a progressive training strategy.
The proposed method is evaluated for the recently proposed VFM, Segment
Anything Model, fine-tuned on segmentation tasks. The searched model yields
about a 95% reduction in BitOPs without incurring performance degradation.

对于大型和高性能的视觉基础模型（Vision Foundation Models，VFMs）进行任意位操作（BitOPs）的压缩，以在各种硬件上部署。我们提出了将 VFM 微调为混合精度量化超网络的方法，该超网络进行神经架构搜索（NAS），可以训练超网络，然后可以提取在任意硬件预算内的子网络。针对现有方法在优化混合精度搜索空间和训练过程中产生大量内存开销方面的困难，我们首先通过比较不同操作符（如分辨率、特征大小、宽度、深度和位宽）的性能和 BitOPs 减少来研究微调 VFM 的有效搜索空间设计。其次，我们提出了一种使用低秩适配器（LoRA）和渐进训练策略的内存高效超网络训练方法。该方法在最近提出的 VFM（Segment Anything Model）上进行了评估，并在分割任务上微调。搜索出的模型在不降低性能的情况下减少了约 95% 的 BitOPs。

使用低秩适配器从视觉基础模型进行混合精度 Supernet 训练

Mixed-precision Supernet Training from Vision Foundation Models using  Low Rank Adapter

We introduce a method to train Quantized Neural Networks (QNNs) --- neural
networks with extremely low precision (e.g., 1-bit) weights and activations, at
run-time. At train-time the quantized weights and activations are used for
computing the parameter gradients. During the forward pass, QNNs drastically
reduce memory size and accesses, and replace most arithmetic operations with
bit-wise operations. As a result, power consumption is expected to be
drastically reduced. We trained QNNs over the MNIST, CIFAR-10, SVHN and
ImageNet datasets. The resulting QNNs achieve prediction accuracy comparable to
their 32-bit counterparts. For example, our quantized version of AlexNet with
1-bit weights and 2-bit activations achieves $51\%$ top-1 accuracy. Moreover,
we quantize the parameter gradients to 6-bits as well which enables gradients
computation using only bit-wise operation. Quantized recurrent neural networks
were tested over the Penn Treebank dataset, and achieved comparable accuracy as
their 32-bit counterparts using only 4-bits. Last but not least, we programmed
a binary matrix multiplication GPU kernel with which it is possible to run our
MNIST QNN 7 times faster than with an unoptimized GPU kernel, without suffering
any loss in classification accuracy. The QNN code is available online.

本研究介绍了一种训练低精度神经网络的方法，该方法使用二进制操作代替计算，达到降低内存大小、减少电力消耗的目的。经过 MNIST、CIFAR-10、SVHN、ImageNet 和 Penn Treebank 数据集的测试，结果表明 1 位权重和 2 位激活的量化版本的 AlexNet 能够达到 51% 的准确率，训练过程中也能使用仅有的二进制操作实现损失函数的计算，并在损失部分的代码上进行了优化，使得 QNN 的运行速度能比未优化的 GPU 加速速度快七倍，并且没有影响分类准确性。