BriefGPT.xyz
May, 2024
SPP:稀疏保存的参数高效微调大型语言模型
SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models
HTML
PDF
Xudong Lu, Aojun Zhou, Yuhui Xu, Renrui Zhang, Peng Gao...
TL;DR
介绍了一种基于稀疏保持参数高效微调的方法,通过轻量级可学习的列和行矩阵对稀疏大语言模型的权重进行优化,保持修剪过的预训练模型的结构和稀疏性,显著提升了稀疏大语言模型的性能。
Abstract
large language models
(LLMs) have become pivotal in advancing the field of artificial intelligence, yet their immense sizes pose significant challenges for both fine-tuning and deployment. Current
post-training pruning
→