Adapting pre-trained foundation models for various downstream tasks has been
prevalent in artificial intelligence. Due to the vast number of tasks and high
costs, adjusting all parameters becomes unfeasible. To mitigate this, several
fine-tuning techniques have been developed to update the pre-trained model
weights in a more resource-efficient manner, such as through low-rank
adjustments. Yet, almost all of these methods focus on linear weights,
neglecting the intricacies of parameter spaces in higher dimensions like 4D.
Alternatively, some methods can be adapted for high-dimensional parameter space
by compressing changes in the original space into two dimensions and then
employing low-rank matrix decomposition. However, these approaches destructs
the structural integrity of the involved high-dimensional spaces. To tackle the
diversity of dimensional spaces across different foundation models and provide
a more precise representation of the changes within these spaces, this paper
introduces a generalized parameter-efficient fine-tuning framework, FLoRA,
designed for various dimensional parameter space. Specifically, utilizing
Tucker decomposition, FLoRA asserts that changes in each dimensional parameter
space are based on a low-rank core space which maintains the consistent
topological structure with the original space. It then models the changes
through this core space alongside corresponding weights to reconstruct
alterations in the original space. FLoRA effectively preserves the structural
integrity of the change of original N-dimensional parameter space, meanwhile
decomposes it via low-rank tensor decomposition. Extensive experiments on
computer vision, natural language processing and multi-modal tasks validate
FLoRA's effectiveness. Codes are available at
this https URL

本文介绍了针对不同维参数空间的一种广义参数高效微调框架 FLoRA，通过 Tucker 分解在保持与原始空间一致的拓扑结构的低秩核空间上建模变化，并将相关权重用于重构原始空间中的变化，从而有效地保留原始 N 维参数空间变化的结构完整性，同时通过低秩张量分解对其进行分解。大量在计算机视觉、自然语言处理和多模态任务上的实验验证了 FLoRA 的有效性。