Fine-tuning techniques based on Large Pretrained Language Models (LPLMs) have
been proven to significantly enhance model performance on a variety of
downstream tasks and effectively control the output behaviors of LPLMs. Recent
studies have proposed numerous methods for fine-tuning a small number of
parameters based on open-source LPLMs, reducing the demand for computational
and storage resources. Among these, reparameterization fine-tuning methods
represented by LoRA (Low-Rank Adaptation) have gained popularity. We find that
although these methods perform well in many aspects, there is still
considerable room for improvement in terms of complex task adaptability,
performance, stability, and algorithm complexity. In response to this, inspired
by the idea that the functions of the brain are shaped by its geometric
structure, this paper integrates this idea into LoRA technology and proposes a
new matrix transformation-based reparameterization method for efficient
fine-tuning, named Matrix-Transformation based Low-Rank Adaptation (MTLoRA).
MTLoRA aims to dynamically alter its spatial geometric structure by applying a
transformation-matrix T to perform linear transformations, such as rotation,
scaling, and translation, on the task-specific parameter matrix, generating new
matrix feature patterns (eigenvectors) to mimic the fundamental influence of
complex geometric structure feature patterns in the brain on functions, thereby
enhancing the model's performance in downstream tasks. In Natural Language
Understanding (NLU) tasks, it is evaluated using the GLUE benchmark test, and
the results reveal that MTLoRA achieves an overall performance increase of
about 1.0% across eight tasks; in Natural Language Generation (NLG) tasks,
MTLoRA improves performance by an average of 0.95% and 0.31% in the DART and
WebNLG tasks, respectively.

该论文介绍了一种基于大规模预训练语言模型 (LPLMs) 的微调技术，通过矩阵变换的重新参数化方法 (MTLoRA) 在下游任务中提高模型性能，取得了显著的提升。