Parameter-efficient fine-tuning approaches have recently garnered a lot of
attention. Having considerably lower number of trainable weights, these methods
can bring about scalability and computational effectiveness. In this paper, we
look for optimal sub-networks and investigate the capability of different
transformer modules in transferring knowledge from a pre-trained model to a
downstream task. Our empirical results suggest that every transformer module in
BERT can act as a winning ticket: fine-tuning each specific module while
keeping the rest of the network frozen can lead to comparable performance to
the full fine-tuning. Among different modules, LayerNorms exhibit the best
capacity for knowledge transfer with limited trainable weights, to the extent
that, with only 0.003% of all parameters in the layer-wise analysis, they show
acceptable performance on various target tasks. On the reasons behind their
effectiveness, we argue that their notable performance could be attributed to
their high-magnitude weights compared to that of the other modules in the
pre-trained BERT.

本文旨在通过 fine-tuning 各种 transform 的模块，找寻最优子网络并探究不同 transformer 模块在将知识从预训练模型传输到下游任务方面的能力，结果显示其中的 LayerNorms 具有最佳的限制可训练参数数量和知识传输能力，并且这种效果可能是由于相对于其他模块，在预训练 BERT 中，它们具有更高的权重大小。