BriefGPT.xyz
Feb, 2024
OneBit: 极低位大型语言模型
OneBit: Towards Extremely Low-bit Large Language Models
HTML
PDF
Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu...
TL;DR
该研究使用1位量化来减少高度期望的低精度模型的存储和计算开销,并通过引入一种1位量化感知训练框架OneBit以及基于矩阵分解的参数初始化方法来实现良好的性能(至少达到非量化性能的83%)。
Abstract
model quantification
uses
low bit-width
values to represent the
weight matrices
of models, which is a promising approach to reduce both st
→