Despite large neural networks demonstrating remarkable abilities to complete
different tasks, they require excessive memory usage to store the optimization
states for training. To alleviate this, the low-rank adaptation (LoRA) is
proposed to reduce the optimization states by training fewer parameters.
However, LoRA restricts overall weight update matrices to be low-rank, limiting
the model performance. In this work, we investigate the dynamics of LoRA and
identify that it can be approximated by a random projection. Based on this
observation, we propose Flora, which is able to achieve high-rank updates by
resampling the projection matrices while enjoying the sublinear space
complexity of optimization states. We conduct experiments across different
tasks and model architectures to verify the effectiveness of our approach.

提出了低秩适应（LoRA）方法，通过训练较少的参数来减少优化状态的存储消耗，但会限制整体权重更新矩阵为低秩，本研究基于观察到低秩适应可以用随机投影来近似，因此提出了 Flora 方法，通过重新采样投影矩阵实现高秩更新，同时享受优化状态的次线性空间复杂度，实验证明了我们方法的有效性。