Why can pre-trained language models (PLMs) learn universal representations
and effectively adapt to broad NLP tasks differing a lot superficially? In this
work, we empirically find evidence indicating that the adaptations of PLMs to
various few-shot tasks can be reparameterized as optimizing only a few free
parameters in a unified low-dimensional intrinsic task subspace, which may help
us understand why PLMs could easily adapt to various NLP tasks with small-scale
data. To find such a subspace and examine its universality, we propose an
analysis pipeline called intrinsic prompt tuning (IPT). Specifically, we resort
to the recent success of prompt tuning and decompose the soft prompts of
multiple NLP tasks into the same low-dimensional nonlinear subspace, then we
learn to adapt the PLM to unseen data or tasks by only tuning parameters in
this subspace. In the experiments, we study diverse few-shot NLP tasks and
surprisingly find that in a 250-dimensional subspace found with 100 tasks, by
only tuning 250 free parameters, we can recover 97% and 83% of the full prompt
tuning performance for 100 seen tasks (using different training data) and 20
unseen tasks, respectively, showing great generalization ability of the found
intrinsic task subspace. Besides being an analysis tool, IPT could further help
us improve the prompt tuning stability.

本文提出了一种称为内在提示调整 (IPT) 的分析流程，用于在一个统一的低维自然语言处理任务子空间内进行自然语言处理 (PLM) 的调整，研究结果表明，在 250 维的子空间中只需调整 250 个自由参数，即可表现出高的通用性。