We present a simple yet novel parameterized form of linear mapping to
achieves remarkable network compression performance: a pseudo SVD called
Ternary SVD (TSVD).
Unlike vanilla SVD, TSVD limits the $U$ and $V$ matrices in SVD to ternary
matrices form in $\{\pm 1, 0\}$. This means that instead of using the expensive
multiplication instructions, TSVD only requires addition instructions when
computing $U(\cdot)$ and $V(\cdot)$.
We provide direct and training transition algorithms for TSVD like Post
Training Quantization and Quantization Aware Training respectively.
Additionally, we analyze the convergence of the direct transition algorithms in
theory.
In experiments, we demonstrate that TSVD can achieve state-of-the-art network
compression performance in various types of networks and tasks, including
current baseline models such as ConvNext, Swim, BERT, and large language model
like OPT.

我们提出了一种简单而新颖的参数化线性映射形式，称为 Ternary SVD（TSVD）。与传统的奇异值分解（SVD）不同，TSVD 限制了 SVD 中的 U 和 V 矩阵为三值矩阵形式（±1，0）。在计算 U（・）和 V（・）时，TSVD 仅需要加法指令，而无需昂贵的乘法指令。我们提供了直接转换算法和训练转换算法的过渡算法（如后训练量化和量化感知训练）。此外，我们理论上分析了直接转换算法的收敛性。在实验证明，TSVD 可以在各种类型的网络和任务中实现最先进的网络压缩性能，包括当前的基线模型，如 ConvNext、Swim、BERT 和大型语言模型 OPT。