BriefGPT.xyz
Feb, 2025
几乎无损的自适应比特切换
Nearly Lossless Adaptive Bit Switching
HTML
PDF
Haiduo Huang, Zhenhua Liu, Tian Xia, Wenzhe zhao, Pengju Ren
TL;DR
本研究解决了模型量化在深度神经网络训练中关于比特宽度不统一的问题,提出了一种新的方法以减少存储成本并优化训练过程。通过引入双舍入量化方法和自适应学习率缩放技术,实现了几乎无损的比特切换,并在多个任务上验证了其有效性,展示了相较于现有技术的显著优势。
Abstract
Model Quantization
is widely applied for compressing and accelerating
Deep Neural Networks
(DNNs). However, conventional Quantization-Aware Training (QAT) focuses on training DNNs with uniform bit-width. The bit-
→