With the increasingly powerful performances and enormous scales of Pretrained
Language Models (PLMs), promoting parameter efficiency in fine-tuning has
become a crucial need for effective and efficient adaptation to various
downstream tasks. One representative line of fine-tuning methods is Orthogonal
Fine-tuning (OFT), which rigorously preserves the angular distances within the
parameter space to preserve the pretrained knowledge. Despite the empirical
effectiveness, OFT still suffers low parameter efficiency at $\mathcal{O}(d^2)$
and limited capability of downstream adaptation. Inspired by Givens rotation,
in this paper, we proposed quasi-Givens Orthogonal Fine-Tuning (qGOFT) to
address the problems. We first use $\mathcal{O}(d)$ Givens rotations to
accomplish arbitrary orthogonal transformation in $SO(d)$ with provable
equivalence, reducing parameter complexity from $\mathcal{O}(d^2)$ to
$\mathcal{O}(d)$. Then we introduce flexible norm and relative angular
adjustments under soft orthogonality regularization to enhance the adaptation
capability of downstream semantic deviations. Extensive experiments on various
tasks and PLMs validate the effectiveness of our methods.

通过引入准 Givens 正交微调（qGOFT），本文旨在提高 Fine-tuning 中的参数效率，并增强其适应各种下游任务的能力。在实验中，我们使用不同的任务和预训练语言模型验证了我们方法的有效性。

参数高效的准正交微调：基于 Givens 旋转方法

Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation

Transformer-based pretrained language models (T-PTLMs) have achieved great
success in almost every NLP task. The evolution of these models started with
GPT and BERT. These models are built on the top of transformers,
self-supervised learning and transfer learning. Transformed-based PTLMs learn
universal language representations from large volumes of text data using
self-supervised learning and transfer this knowledge to downstream tasks. These
models provide good background knowledge to downstream tasks which avoids
training of downstream models from scratch. In this comprehensive survey paper,
we initially give a brief overview of self-supervised learning. Next, we
explain various core concepts like pretraining, pretraining methods,
pretraining tasks, embeddings and downstream adaptation methods. Next, we
present a new taxonomy of T-PTLMs and then give brief overview of various
benchmarks including both intrinsic and extrinsic. We present a summary of
various useful libraries to work with T-PTLMs. Finally, we highlight some of
the future research directions which will further improve these models. We
strongly believe that this comprehensive survey paper will serve as a good
reference to learn the core concepts as well as to stay updated with the recent
happenings in T-PTLMs.

本论文对基于 Transformer 的预训练语言模型进行了全面调查，并介绍了自监督学习、预训练方法、嵌入、下游适应方法等核心概念以及 T-PTLMs 的新分类法，提供了各种有用的库以及未来研究方向，该论文可作为学习核心概念和了解 T-PTLMs 近期进展的参考资料。