Large Language Models (LLMs) have dramatically advanced AI applications, yet
their deployment remains challenging due to their immense inference costs.
Recent studies ameliorate the computational costs of LLMs by increasing their
activation sparsity but suffer from significant performance degradation on
downstream tasks. In this work, we introduce a new framework for sparsifying
the activations of base LLMs and reducing inference costs, dubbed Contextually
Aware Thresholding for Sparsity (CATS). CATS is relatively simple, easy to
implement, and highly effective. At the heart of our framework is a new
non-linear activation function. We demonstrate that CATS can be applied to
various base models, including Mistral-7B and Llama2-7B, and outperforms
existing sparsification techniques in downstream task performance. More
precisely, CATS-based models often achieve downstream task performance within
1-2% of their base models without any fine-tuning and even at activation
sparsity levels of 50%. Furthermore, CATS-based models converge faster and
display better task performance than competing techniques when fine-tuning is
applied. Finally, we develop a custom GPU kernel for efficient implementation
of CATS that translates the activation of sparsity of CATS to real wall-clock
time speedups. Our custom kernel implementation of CATS results in a ~15%
improvement in wall-clock inference latency of token generation on both
Llama-7B and Mistral-7B.

基于 Contextually Aware Thresholding for Sparsity (CATS) 的新框架用于稀疏化大型语言模型的激活并降低推理成本，在不需要微调的情况下，CATS-based 模型可以在 50% 的激活稀疏程度下实现与基础模型 1-2% 的下游任务性能，并且具有更快的收敛速度和更好的任务性能。