The increasingly Large Language Models (LLMs) demonstrate stronger language
understanding and generation capabilities, while the memory demand and
computation cost of fine-tuning LLMs on downstream tasks are non-negligible.
Besides, fine-tuning generally requires a certain amount of data from
individual tasks whilst data collection cost is another issue to consider in
real-world applications. In this work, we focus on Parameter-Efficient
Fine-Tuning (PEFT) methods for few-shot Natural Language Generation (NLG),
which freeze most parameters in LLMs and tune a small subset of parameters in
few-shot cases so that memory footprint, training cost, and labeling cost are
reduced while maintaining or even improving the performance. We propose a
Scaled Prompt-Tuning (SPT) method which surpasses conventional PT with better
performance and generalization ability but without an obvious increase in
training cost. Further study on intermediate SPT suggests the superior
transferability of SPT in few-shot scenarios, providing a recipe for
data-deficient and computation-limited circumstances. Moreover, a comprehensive
comparison of existing PEFT methods reveals that certain approaches exhibiting
decent performance with modest training cost such as Prefix-Tuning in prior
study could struggle in few-shot NLG tasks, especially on challenging datasets.

本文研究了参数高效微调方法和少样本自然语言生成，提出了一个超越传统方法且在训练成本上没有明显增加的方法，同时通过中间实验证明了该方法在少样本场景中的卓越可迁移性，为数据不充足和计算有限的情况提供了解决方案。此外，综合比较了多种参数高效微调方法，揭示了在少样本自然语言生成任务中某些方法在挑战性数据集上可能存在困难。