Quantization emerges as one of the most promising compression technologies
for deploying efficient large models for various real time application in
recent years. Considering that the storage and IO of weights take up the vast
majority of the overhead inside a large model, weight only quantization can
lead to large gains. However, existing quantization schemes suffer from
significant accuracy degradation at very low bits, or require some additional
computational overhead when deployed, making it difficult to be applied to
large-scale applications in industry. In this paper, we propose decoupleQ,
achieving a substantial increase in model accuracy, especially at very low
bits. decoupleQ abandons the traditional heuristic quantization paradigm and
decouples the model parameters into integer and floating-point parts, thus
transforming the quantization problem into a traditional mathematical
optimization problem with constraints, which is then solved alternatively by
off-the-shelf optimization methods.
Quantization via decoupleQ is linear and uniform, making it
hardware-friendlier than non-uniform counterpart, and enabling the idea to be
migrated to high-bit quantization to enhance its robustness. Our method has
achieved well on-line accuracy near fp16/bf16 on the 2-bit quantization of
large speech models in ByteDance. The code is available at
this https URL

通过解耦 Q，我们提出了一种量化方法，将模型参数分解为整数部分和浮点部分，使得量化问题转化为约束条件下的数学优化问题，并通过现成的优化方法解决。该方法在线上实现了接近 fp16/bf16 准确度的 2 位量化大型语音模型，在硬件上更友好且能够应用于高位量化以增强其鲁棒性。

decoupleQ：通过将参数拆分为整数和浮点数实现 2-bit 后训练统一量化

decoupleQ: Towards 2-bit Post-Training Uniform Quantization via  decoupling Parameters into Integer and Floating Points

Knowledge Distillation (KD) emerges as one of the most promising compression
technologies to run advanced deep neural networks on resource-limited devices.
In order to train a small network (student) under the guidance of a large
network (teacher), the intuitive method is regularizing the feature maps or
logits of the student using the teacher's information. However, existing
methods either over-restrict the student to learn all information from the
teacher, which lead to some bad local minimum, or use various fancy and
elaborate modules to process and align features, which are complex and lack
generality. In this work, we proposed an abstract and general paradigm for the
KD task, referred to as DIMensionality Reduction KD (RdimKD), which solely
relies on dimensionality reduction, with a very minor modification to naive L2
loss. RdimKD straightforwardly utilizes a projection matrix to project both the
teacher's and student's feature maps onto a low-dimensional subspace, which are
then optimized during training. RdimKD achieves the goal in the simplest way
that not only does the student get valuable information from the teacher, but
it also ensures sufficient flexibility to adapt to the student's low-capacity
reality. Our extensive empirical findings indicate the effectiveness of RdimKD
across various learning tasks and diverse network architectures.

这篇研究提出了一种新的知识蒸馏范式，称为维度降低知识蒸馏（RdimKD），通过使用一个投影矩阵将大网络和小网络的特征图投影到低维子空间，并在训练过程中进行优化，既保证学生从老师获取有价值信息，又灵活适应学生的低容量实际情况。实证研究表明 RdimKD 在各种学习任务和不同网络架构中具有有效性。