This paper explores a new post-hoc training-free compression paradigm for
compressing Large Language Models (LLMs) to facilitate their wider adoption in
various computing environments. We delve into the challenges of LLM
compression, notably their dependency on extensive training data and
computational resources. We propose a training-free approach dubbed
Activation-aware Singular Value Decomposition (ASVD) to address these
limitations. ASVD effectively manages activation outliers by adjusting the
weight matrix based on the activation distribution, improving decomposition
accuracy and efficiency. Our method also addresses the varying sensitivity of
different LLM layers to decomposition, with an iterative calibration process
for optimal layer-specific decomposition. Experiments demonstrate that ASVD can
compress network by 10%-20% without losing reasoning capacities. Additionally,
it can be seamlessly integrated with other LLM compression paradigms,
showcasing its flexible compatibility. Code and compressed models are available
at this https URL

这篇论文研究了一种新的后期无训练压缩范式，用于在各种计算环境中更广泛地采用压缩大型语言模型（LLMs）。我们探讨了 LLM 压缩的挑战，特别是它们对广泛的训练数据和计算资源的依赖。我们提出了一种称为激活感知奇异值分解（ASVD）的无训练方法来解决这些限制。ASVD 通过根据激活分布调整权重矩阵来有效管理激活异常值，提高分解的准确性和效率。我们的方法还解决了不同 LLM 层对分解的不同敏感性问题，通过迭代校准过程实现最佳的层特定分解。实验证明，ASVD 能够在不损失推理能力的情况下压缩网络 10%-20%。此外，它可以与其他 LLM 压缩范式无缝集成，展示了它的灵活兼容性。代码和压缩模型可以在此网址找到。