Given the generational gap in available hardware between lay practitioners
and the most endowed institutions, LLMs are becoming increasingly inaccessible
as they grow in size. Whilst many approaches have been proposed to compress
LLMs to make their resource consumption manageable, these methods themselves
tend to be resource intensive, putting them out of the reach of the very user
groups they target. In this work, we explore the problem of structured pruning
of LLMs using only forward passes. We seek to empower practitioners to prune
models so large that their available hardware has just enough memory to run
inference. We develop Bonsai, a gradient-free, perturbative pruning method
capable of delivering small, fast, and accurate pruned models.
We observe that Bonsai outputs pruned models that (i) outperform those
generated by more expensive gradient-based structured pruning methods, and (ii)
are twice as fast (with comparable accuracy) as those generated by
semi-structured pruning methods requiring comparable resources as Bonsai. We
also leverage Bonsai to produce a new sub-2B model using a single A6000 that
yields state-of-the-art performance on 4/6 tasks on the Huggingface Open LLM
leaderboard.

利用仅使用前向传递的方法进行结构裁剪，我们开发了 Bonsai，一种无梯度、扰动性的裁剪方法，能够生成小型、快速、准确的模型，超越梯度裁剪方法和半结构裁剪方法的性能且节约资源。

每个人都剪枝：只使用前向传播的 LLM 结构剪枝

Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes

Extreme multi-label classification (XMC) refers to supervised multi-label
learning involving hundreds of thousand or even millions of labels. In this
paper, we develop a suite of algorithms, called Bonsai, which generalizes the
notion of label representation in XMC, and partitions the labels in the
representation space to learn shallow trees. We show three concrete
realizations of this label representation space including : (i) the input space
which is spanned by the input features, (ii) the output space spanned by label
vectors based on their co-occurrence with other labels, and (iii) the joint
space by combining the input and output representations. Furthermore, the
constraint-free multi-way partitions learnt iteratively in these spaces lead to
shallow trees. By combining the effect of shallow trees and generalized label
representation, Bonsai achieves the best of both worlds - fast training which
is comparable to state-of-the-art tree-based methods in XMC, and much better
prediction accuracy, particularly on tail-labels. On a benchmark Amazon-3M
dataset with 3 million labels, \bonsai outperforms a state-of-the-art
one-vs-rest method in terms of prediction accuracy, while being approximately
200 times faster to train. The code for Bonsai is available at
https://github.com/xmc-aalto/bonsai

本文研究了极端多标签分类问题，提出了一种基于标签表示和分区的算法集 Bonsai，通过生成浅层树结构实现了快速训练和更好的预测精度，特别是在较少出现的小众标签方面取得了显著的效果提升。在 Amazon-3M 数据集上对比已有方法进行实验，结果表明 Bonsai 在预测效果上优于现有方法，并且训练速度快 200 倍以上。