Adapting large pre-trained models (PTMs) through fine-tuning imposes
prohibitive computational and storage burdens. Recent studies of
parameter-efficient tuning (PET) find that only optimizing a small portion of
parameters conditioned on PTMs could yield on-par performance compared to
conventional fine-tuning. Generally, PET methods exquisitely design
parameter-efficient modules (PET modules) which could be applied to arbitrary
fine-grained positions inside PTMs. However, the effectiveness of these
fine-grained positions largely relies on sophisticated manual designation,
thereby usually producing sub-optimal results. In contrast to the manual
designation, we explore constructing PET modules in an automatic manner. We
automatically \textbf{S}earch for the \textbf{S}parse \textbf{S}tructure of
\textbf{P}arameter-\textbf{E}fficient \textbf{T}uning (S$^3$PET). Based on a
unified framework of various PET methods, S$^3$PET conducts the differentiable
PET structure search through bi-level optimization and proposes shifted global
sigmoid method to explicitly control the number of trainable parameters.
Extensive experiments show that S$^3$PET surpasses manual and random structures
with less trainable parameters. The searched structures preserve more than 99\%
fine-tuning performance with 0.01\% trainable parameters. Moreover, the
advantage of S$^3$PET is amplified with extremely low trainable parameters
budgets (0.0009\%$\sim$0.01\%). The searched structures are transferable and
explainable, providing suggestions and guidance for the future design of PET
methods.

通过自动化的 PET 模块结构搜索方法 S^3PET，我们能够以远远低于用于传统模型的可训练参数的预算对预训练模型进行微调，搜索得到的结构具有高度可迁移性和可解释性且在保留超过 99% 微调性能的同时，只需 0.01% 的可训练参数。