BriefGPT.xyz
Jun, 2024
预训练变换器的矩阵分解方法的实证研究
An Empirical Investigation of Matrix Factorization Methods for Pre-trained Transformers
HTML
PDF
Ashim Gupta, Sina Mahdipour Saravani, P. Sadayappan, Vivek Srikumar
TL;DR
通过比较传统的低秩分解与最近引入的 Monarch 分解,我们在 Transformer 压缩技术方面提出了一个全面的分析,发现低秩分解在不同压缩比和六个文本分类任务中始终优于 Monarch 分解。
Abstract
The increasing size of
transformer-based models
in NLP makes the question of compressing them important. In this work, we present a comprehensive analysis of factorization based
model compression techniques
. Spec
→