We present GIFT (Generative Interpretable Fine-tuning Transformers) for
fine-tuning pretrained (often large) Transformer models at downstream tasks in
a parameter-efficient way with built-in interpretability. Our GIFT is a deep
parameter-residual learning method, which addresses two problems in fine-tuning
a pretrained Transformer model: Where to apply the parameter-efficient
fine-tuning (PEFT) to be extremely lightweight yet sufficiently expressive, and
How to learn the PEFT to better exploit the knowledge of the pretrained model
in a direct way? For the former, we select the final projection (linear) layer
in the multi-head self-attention of a Transformer model, and verify its
effectiveness. For the latter, in contrast to the prior art that directly
introduce new model parameters (often in low-rank approximation form) to be
learned in fine-tuning with downstream data, we propose a method for learning
to generate the fine-tuning parameters. Our GIFT is a hyper-Transformer which
take as input the pretrained parameters of the projection layer to generate its
fine-tuning parameters using a proposed Parameter-to-Cluster Attention (PaCa).
The PaCa results in a simple clustering-based forward explainer that plays the
role of semantic segmentation in testing. In experiments, our proposed GIFT is
tested on the VTAB benchmark and the fine-grained visual classification (FGVC)
benchmark. It obtains significantly better performance than the prior art. Our
code is available at this https URL

GIFT 是一种生成可解释化微调 Transformers 预训练模型的方法，在参数高效的情况下提供内置可解释性，通过选择 Transformer 模型的多头自注意力中的最终投影（线性）层进行 fine-tuning，提出了使用参数到聚类注意力来生成 fine-tuning 参数的超级 Transformer 方法，并在 VTAB 和 FGVC 基准测试中达到了显著更好的性能。