Prompt tuning prepends a soft prompt to the input embeddings or hidden states
and only optimizes the prompt to adapt pretrained models (PTMs) to downstream
tasks. The previous work manually selects prompt layers which are far from
optimal and failed to exploit the potential of prompt tuning. In this work, we
propose a novel framework, \underline{S}elective \underline{P}rompt
\underline{T}uning (SPT), that learns to select the proper prompt layers by
inserting a prompt controlled by a learnable probabilistic gate at each
intermediate layer. We further propose a novel bi-level optimization framework,
SPT-DARTS, that can better optimize the learnable gates and improve the final
prompt tuning performances of the learned prompt layer settings. We conduct
extensive experiments with ten benchmark datasets under the full-data and
few-shot scenarios. The results demonstrate that our SPT framework can perform
better than the previous state-of-the-art PETuning baselines with comparable or
fewer tunable parameters.

本文提出了一种新颖的框架，Selective Prompt Tuning (SPT)，通过在每个中间层插入由可学习的概率门控制的提示来学习选择适当的提示层，进一步提出了一种新颖的双层优化框架 SPT-DARTS，可以更好地优化可学习门并改善所学提示层设置的最终提示调整性能，实验证明我们的 SPT 框架在全数据和少样本情况下比以往的最先进的 PETuning 基准方法表现更好，参数可调性相当或更少。

借助学习的提示层改进提示调优

Improving Prompt Tuning with Learned Prompting Layers

Successful recurrent models such as long short-term memories (LSTMs) and
gated recurrent units (GRUs) use ad hoc gating mechanisms. Empirically these
models have been found to improve the learning of medium to long term temporal
dependencies and to help with vanishing gradient issues. We prove that
learnable gates in a recurrent model formally provide quasi- invariance to
general time transformations in the input data. We recover part of the LSTM
architecture from a simple axiomatic approach. This result leads to a new way
of initializing gate biases in LSTMs and GRUs. Ex- perimentally, this new
chrono initialization is shown to greatly improve learning of long term
dependencies, with minimal implementation effort.

本文研究证明可学习门控在循环神经网络中提供输入数据的时间转换几乎不变性的功能，从而为长期依赖性和参数初始化提供了新的可能性。