In this paper we generalize and extend an idea of low-rank adaptation (LoRA) of large language models (LLMs) based on Transformer architecture. Widely used LoRA-like methods of fine-tuning LLMs are based on matrix factorization of gradient update. We introduce LoTR, a novel approach for parameter-efficient fine-tuning of LLMs which represents a gradient update to parameters in a form of tensor decomposition. Low-rank adapter for each layer is constructed as a product of three matrices, and tensor structure arises from sharing left and right multipliers of this product among layers. Simultaneous compression of a sequence of layers with low-rank tensor representation allows LoTR to archive even better parameter efficiency then LoRA especially for deep models. Moreover, the core tensor does not depend on original weight dimension and can be made arbitrary small, which allows for extremely cheap and fast downstream fine-tuning.

本文提出了一种新颖的参数高效微调方法LoTR，通过张量分解的形式对参数进行梯度更新，进而构建每层的低秩适配器。LoTR在低秩张量表示下对一系列层进行压缩，尤其适用于深度模型，相比于LoRA具有更好的参数效率，且核心张量不依赖于原始权重维度，可以变得任意小，可实现极为廉价快速的下游微调。

LoTR：低张量秩权重适应