Parameter-Efficient Fine-Tuning (PEFT) has become the standard for
customising Foundation Models (FMs) to user-specific downstream tasks. However,
typical PEFT methods require storing multiple task-specific adapters, creating
scalability issues as these adapters must be housed and run at the FM server.
Traditional prompt tuning offers a potential solution by customising them
through task-specific input prefixes, but it under-performs compared to other
PEFT methods like LoRA. To address this gap, we propose Low-Rank Prompt
Adaptation (LOPA), a prompt-tuning-based approach that performs on par with
state-of-the-art PEFT methods and full fine-tuning while being more
parameter-efficient and not requiring a server-based adapter. LOPA generates
soft prompts by balancing between sharing task-specific information across
instances and customization for each instance. It uses a low-rank decomposition
of the soft-prompt component encoded for each instance to achieve parameter
efficiency. We provide a comprehensive evaluation on multiple natural language
understanding and code generation and understanding tasks across a wide range
of foundation models with varying sizes.

通过使用 “Low-Rank Prompt Adaptation”，我们提出了一种基于 prompt 调优的方法，它在可接受的参数效率下与最先进的 PEFT 方法和全面调优方法具有相当的性能，而且无需基于服务器的适配器。