When pre-trained models become rapidly larger, the cost of fine-tuning on
downstream tasks steadily increases, too. To economically fine-tune these
models, parameter-efficient transfer learning (PETL) is proposed, which only
tunes a tiny subset of trainable parameters to efficiently learn quality
representations. However, current PETL methods are facing the dilemma that
during training the GPU memory footprint is not effectively reduced as
trainable parameters. PETL will likely fail, too, if the full fine-tuning
encounters the out-of-GPU-memory issue. This phenomenon happens because
trainable parameters from these methods are generally entangled with the
backbone, such that a lot of intermediate states have to be stored in GPU
memory for gradient propagation. To alleviate this problem, we introduce
Disentangled Transfer Learning (DTL), which disentangles the trainable
parameters from the backbone using a lightweight Compact Side Network (CSN). By
progressively extracting task-specific information with a few low-rank linear
mappings and appropriately adding the information back to the backbone, CSN
effectively realizes knowledge transfer in various downstream tasks. We
conducted extensive experiments to validate the effectiveness of our method.
The proposed method not only reduces a large amount of GPU memory usage and
trainable parameters, but also outperforms existing PETL methods by a
significant margin in accuracy, achieving new state-of-the-art on several
standard benchmarks.

通过提出深度解耦迁移学习（DTL），我们可以减少大量的 GPU 内存使用和可训练参数，并在准确性上显著超过现有的 PETL 方法，在几个标准基准上达到了新的最先进水平。