Parameter-efficient tuning (PET) has been widely explored in recent years
because it tunes much fewer parameters (PET modules) than full-parameter
fine-tuning (FT) while still stimulating sufficient knowledge from large
language models (LLMs) for downstream tasks. Moreover, when PET is employed to
serve multiple tasks, different task-specific PET modules can be built on a
frozen LLM, avoiding redundant LLM deployments. Although PET significantly
reduces the cost of tuning and deploying LLMs, its inference still suffers from
the computational bottleneck of LLMs. To address the above issue, we propose an
effective PET framework based on compressed LLMs, named "CPET". In CPET, we
evaluate the impact of mainstream LLM compression techniques on PET performance
and then introduce knowledge inheritance and recovery strategies to restore the
knowledge loss caused by these compression techniques. Our experimental results
demonstrate that, owing to the restoring strategies of CPET, collaborating
task-specific PET modules with a compressed LLM can achieve comparable
performance to collaborating PET modules with the original version of the
compressed LLM and outperform directly applying vanilla PET methods to the
compressed LLM.

本文提出了一个基于压缩语言模型的参数有效调整框架，名为 CPET。CPET 在评估了 LLM 压缩技术对 PET 性能的影响后，介绍了知识继承和恢复策略来恢复由于这些压缩技术引起的知识丢失。实验结果表明，与原始 LLM 相比，与压缩 LLM 合作的特定任务 PET 模块可以实现可比的性能并且优于直接将基本 PET 方法应用于压缩 LLM。

CPET：压缩大型语言模型有效参数高效调整

CPET: Effective Parameter-Efficient Tuning for Compressed Large Language  Models

Adapting large pre-trained models (PTMs) through fine-tuning imposes
prohibitive computational and storage burdens. Recent studies of
parameter-efficient tuning (PET) find that only optimizing a small portion of
parameters conditioned on PTMs could yield on-par performance compared to
conventional fine-tuning. Generally, PET methods exquisitely design
parameter-efficient modules (PET modules) which could be applied to arbitrary
fine-grained positions inside PTMs. However, the effectiveness of these
fine-grained positions largely relies on sophisticated manual designation,
thereby usually producing sub-optimal results. In contrast to the manual
designation, we explore constructing PET modules in an automatic manner. We
automatically \textbf{S}earch for the \textbf{S}parse \textbf{S}tructure of
\textbf{P}arameter-\textbf{E}fficient \textbf{T}uning (S$^3$PET). Based on a
unified framework of various PET methods, S$^3$PET conducts the differentiable
PET structure search through bi-level optimization and proposes shifted global
sigmoid method to explicitly control the number of trainable parameters.
Extensive experiments show that S$^3$PET surpasses manual and random structures
with less trainable parameters. The searched structures preserve more than 99\%
fine-tuning performance with 0.01\% trainable parameters. Moreover, the
advantage of S$^3$PET is amplified with extremely low trainable parameters
budgets (0.0009\%$\sim$0.01\%). The searched structures are transferable and
explainable, providing suggestions and guidance for the future design of PET
methods.

通过自动化的 PET 模块结构搜索方法 S^3PET，我们能够以远远低于用于传统模型的可训练参数的预算对预训练模型进行微调，搜索得到的结构具有高度可迁移性和可解释性且在保留超过 99% 微调性能的同时，只需 0.01% 的可训练参数。