We present GIFT (Generative Interpretable Fine-tuning Transformers) for
fine-tuning pretrained (often large) Transformer models at downstream tasks in
a parameter-efficient way with built-in interpretability. Our GIFT is a deep
parameter-residual learning method, which addresses two problems in fine-tuning
a pretrained Transformer model: Where to apply the parameter-efficient
fine-tuning (PEFT) to be extremely lightweight yet sufficiently expressive, and
How to learn the PEFT to better exploit the knowledge of the pretrained model
in a direct way? For the former, we select the final projection (linear) layer
in the multi-head self-attention of a Transformer model, and verify its
effectiveness. For the latter, in contrast to the prior art that directly
introduce new model parameters (often in low-rank approximation form) to be
learned in fine-tuning with downstream data, we propose a method for learning
to generate the fine-tuning parameters. Our GIFT is a hyper-Transformer which
take as input the pretrained parameters of the projection layer to generate its
fine-tuning parameters using a proposed Parameter-to-Cluster Attention (PaCa).
The PaCa results in a simple clustering-based forward explainer that plays the
role of semantic segmentation in testing. In experiments, our proposed GIFT is
tested on the VTAB benchmark and the fine-grained visual classification (FGVC)
benchmark. It obtains significantly better performance than the prior art. Our
code is available at this https URL

GIFT 是一种生成可解释化微调 Transformers 预训练模型的方法，在参数高效的情况下提供内置可解释性，通过选择 Transformer 模型的多头自注意力中的最终投影（线性）层进行 fine-tuning，提出了使用参数到聚类注意力来生成 fine-tuning 参数的超级 Transformer 方法，并在 VTAB 和 FGVC 基准测试中达到了显著更好的性能。

GIFT: 生成解释性微调变压器

GIFT: Generative Interpretable Fine-Tuning Transformers

We focus on the problem of domain adaptation when the goal is shifting the
model towards the target distribution, rather than learning domain invariant
representations. It has been shown that under the following two assumptions:
(a) access to samples from intermediate distributions, and (b) samples being
annotated with the amount of change from the source distribution, self-training
can be successfully applied on gradually shifted samples to adapt the model
toward the target distribution. We hypothesize having (a) is enough to enable
iterative self-training to slowly adapt the model to the target distribution,
by making use of an implicit curriculum. In the case where (a) does not hold,
we observe that iterative self-training falls short. We propose GIFT, a method
that creates virtual samples from intermediate distributions by interpolating
representations of examples from source and target domains. We evaluate an
iterative-self-training method on datasets with natural distribution shifts,
and show that when applied on top of other domain adaptation methods, it
improves the performance of the model on the target dataset. We run an analysis
on a synthetic dataset to show that in the presence of (a)
iterative-self-training naturally forms a curriculum of samples. Furthermore,
we show that when (a) does not hold, GIFT performs better than iterative
self-training.

本研究提出了一种自适应方法 GIFT，它利用从源域和目标域的例子的表示进行插值，从而在自适应过程中创建中间分布的虚拟样本。同时，GIFT 在自适应中生成一个样本的教学计划来逐步适应目标分布。