Context-based fine-tuning methods, including prompting, in-context learning,
soft prompting (also known as prompt tuning), and prefix-tuning, have gained
popularity due to their ability to often match the performance of full
fine-tuning with a fraction of the parameters. Despite their empirical
successes, there is little theoretical understanding of how these techniques
influence the internal computation of the model and their expressiveness
limitations. We show that despite the continuous embedding space being more
expressive than the discrete token space, soft-prompting and prefix-tuning are
strictly less expressive than full fine-tuning, even with the same number of
learnable parameters. Concretely, context-based fine-tuning cannot change the
relative attention pattern over the content and can only bias the outputs of an
attention layer in a fixed direction. This suggests that while techniques like
prompting, in-context learning, soft prompting, and prefix-tuning can
effectively elicit skills present in the pretrained model, they cannot learn
novel tasks that require new attention patterns.

基于上下文的微调方法，包括提示、上下文学习、软提示（也称为提示微调）和前缀微调，由于能够在参数的一小部分范围内经常达到完全微调的性能，因此受到了广泛的关注。尽管这些方法在实证上取得了成功，但对于这些技术如何影响模型的内部计算和它们的表达能力限制，在理论上还知之甚少。我们证明了尽管连续的嵌入空间比离散的标记空间更具表达能力，但软提示和前缀微调与完全微调相比，在相同数量的可学习参数下是严格不具表达能力的。具体而言，基于上下文的微调不能改变内容的相对注意模式，并且只能将注意层的输出朝一个固定的方向偏置。这表明，虽然提示、上下文学习、软提示和前缀微调等技术可以有效地引出预训练模型中存在的技能，但它们无法学习需要新的注意模式的新任务。