Vision transformers have revolutionized computer vision, but their
computational demands present challenges for training and deployment. This
paper introduces LOTUS (LOttery Transformers with Ultra Sparsity), a novel
method that leverages data lottery ticket selection and sparsity pruning to
accelerate vision transformer training while maintaining accuracy. Our approach
focuses on identifying and utilizing the most informative data subsets and
eliminating redundant model parameters to optimize the training process.
Through extensive experiments, we demonstrate the effectiveness of LOTUS in
achieving rapid convergence and high accuracy with significantly reduced
computational requirements. This work highlights the potential of combining
data selection and sparsity techniques for efficient vision transformer
training, opening doors for further research and development in this area.

LOTUS 是一种利用数据抽奖选择和稀疏剪枝来加速视觉变换器训练的新方法，通过识别和利用最信息丰富的数据子集并消除冗余模型参数来优化训练过程，通过广泛的实验证明了 LOTUS 在快速收敛和高准确度达到显著降低计算要求方面的有效性，这项工作突出了在高效视觉变换器训练方面结合数据选择和稀疏技术的潜力，为进一步的研究和开发提供了机会。

LOTUS: 通过稀疏修剪和数据抽奖提高 Transformer 效率

LOTUS: Improving Transformer Efficiency with Sparsity Pruning and Data  Lottery Tickets

Various Large Language Models(LLMs) from the Generative Pretrained
Transformer~(GPT) family have achieved outstanding performances in a wide range
of text generation tasks. However, the enormous model sizes have hindered their
practical use in real-world applications due to high inference latency.
Therefore, improving the efficiencies of LLMs through quantization, pruning,
and other means has been a key issue in LLM studies. In this work, we propose a
method based on Hessian sensitivity-aware mixed sparsity pruning to prune LLMs
to at least 50\% sparsity without the need of any retraining. It allocates
sparsity adaptively based on sensitivity, allowing us to reduce pruning-induced
error while maintaining the overall sparsity level. The advantages of the
proposed method exhibit even more when the sparsity is extremely high.
Furthermore, our method is compatible with quantization, enabling further
compression of LLMs.

通过基于 Hessian 灵敏度感知的混合稀疏剪枝方法，我们提出了一种剪枝 LLMs 的方法，以至少达到 50% 的稀疏度，而不需要任何重新训练，该方法适应性地分配稀疏度，减少了剪枝引起的错误，同时保持了整体稀疏度水平，并且在稀疏度极高时表现出更显著的优势，此外，我们的方法与量化兼容，从而进一步压缩 LLMs。