We present Generalized LoRA (GLoRA), an advanced approach for universal
parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA),
GLoRA employs a generalized prompt module to optimize pre-trained model weights
and adjust intermediate activations, providing more flexibility and capability
across diverse tasks and datasets. Moreover, GLoRA facilitates efficient
parameter adaptation by employing a scalable, modular, layer-wise structure
search that learns individual adapter of each layer. Originating from a unified
mathematical formulation, GLoRA exhibits strong transfer learning, few-shot
learning and domain generalization abilities, as it adjusts to new tasks
through additional dimensions on weights and activations. Comprehensive
experiments demonstrate that GLoRA outperforms all previous methods in natural,
specialized, and structured benchmarks, achieving superior accuracy with fewer
parameters and computations on various datasets. Furthermore, our structural
re-parameterization design ensures that GLoRA incurs no extra inference cost,
rendering it a practical solution for resource-limited applications. Code is
available at: this https URL

GLoRA 是一种用于通用参数节约的微调任务的高级方法。它使用广义提示模块来优化预训练模型权重和调整中间激活，并通过采用可伸缩的、模块化的逐层结构搜索来促进高效参数适应，从而实现强大的转移学习、少样本学习和领域泛化能力。