The impressive performance of Large Language Models (LLMs) across various
natural language processing tasks comes at the cost of vast computational
resources and storage requirements. One-shot pruning techniques offer a way to
alleviate these burdens by removing redundant weights without the need for
retraining. Yet, the massive scale of LLMs often forces current pruning
approaches to rely on heuristics instead of optimization-based techniques,
potentially resulting in suboptimal compression. In this paper, we introduce
ALPS, an optimization-based framework that tackles the pruning problem using
the operator splitting technique and a preconditioned conjugate gradient-based
post-processing step. Our approach incorporates novel techniques to accelerate
and theoretically guarantee convergence while leveraging vectorization and GPU
parallelism for efficiency. ALPS substantially outperforms state-of-the-art
methods in terms of the pruning objective and perplexity reduction,
particularly for highly sparse models. On the OPT-30B model with 70% sparsity,
ALPS achieves a 13% reduction in test perplexity on the WikiText dataset and a
19% improvement in zero-shot benchmark performance compared to existing
methods.

本文介绍了一种名为 ALPS 的基于优化的框架，用于通过操作拆分技术和预条件共轭梯度后处理步骤处理修剪问题，并结合向量化和 GPU 并行性以提高效率，在修剪目标和困惑度降低方面远远超过现有方法，特别是对于高度稀疏的模型。在 OPT-30B 模型上，其稀疏度为 70％，ALPS 在 WikiText 数据集上实现了 13％的测试困惑度减少和比现有方法提高了 19％的零样本基准性能。

ALPS：面向大型语言模型的高度稀疏一次性修剪的改进优化

ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large  Language Models

Recent researches have suggested that the predictive accuracy of neural
network may contend with its adversarial robustness. This presents challenges
in designing effective regularization schemes that also provide strong
adversarial robustness. Revisiting Vicinal Risk Minimization (VRM) as a
unifying regularization principle, we propose Adversarial Labelling of
Perturbed Samples (ALPS) as a regularization scheme that aims at improving the
generalization ability and adversarial robustness of the trained model. ALPS
trains neural networks with synthetic samples formed by perturbing each
authentic input sample towards another one along with an adversarially assigned
label. The ALPS regularization objective is formulated as a min-max problem, in
which the outer problem is minimizing an upper-bound of the VRM loss, and the
inner problem is L$_1$-ball constrained adversarial labelling on perturbed
sample. The analytic solution to the induced inner maximization problem is
elegantly derived, which enables computational efficiency. Experiments on the
SVHN, CIFAR-10, CIFAR-100 and Tiny-ImageNet datasets show that the ALPS has a
state-of-the-art regularization performance while also serving as an effective
adversarial training scheme.

本文提出了一种称为 ALPS 的正则化方案，通过向每个真实输入样本添加扰动样本并给予对抗性标签来训练神经网络，以提高其泛化能力和对抗鲁棒性，实验表明 ALPS 具有先进的正则化性能和有效的对抗性训练。