The increasing size of transformer-based models in NLP makes the question of
compressing them important. In this work, we present a comprehensive analysis
of factorization based model compression techniques. Specifically, we focus on
comparing straightforward low-rank factorization against the recently
introduced Monarch factorization, which exhibits impressive performance
preservation on the GLUE benchmark. To mitigate stability issues associated
with low-rank factorization of the matrices in pre-trained transformers, we
introduce a staged factorization approach wherein layers are factorized one by
one instead of being factorized simultaneously. Through this strategy we
significantly enhance the stability and reliability of the compression process.
Further, we introduce a simple block-wise low-rank factorization method, which
has a close relationship to Monarch factorization. Our experiments lead to the
surprising conclusion that straightforward low-rank factorization consistently
outperforms Monarch factorization across both different compression ratios and
six different text classification tasks.

通过比较传统的低秩分解与最近引入的 Monarch 分解，我们在 Transformer 压缩技术方面提出了一个全面的分析，发现低秩分解在不同压缩比和六个文本分类任务中始终优于 Monarch 分解。