Fine-tuning pre-trained models has recently yielded remarkable performance
gains in graph neural networks (GNNs). In addition to pre-training techniques,
inspired by the latest work in the natural language fields, more recent work
has shifted towards applying effective fine-tuning approaches, such as
parameter-efficient tuning (delta tuning). However, given the substantial
differences between GNNs and transformer-based models, applying such approaches
directly to GNNs proved to be less effective. In this paper, we present a
comprehensive comparison of delta tuning techniques for GNNs and propose a
novel delta tuning method specifically designed for GNNs, called AdapterGNN.
AdapterGNN preserves the knowledge of the large pre-trained model and leverages
highly expressive adapters for GNNs, which can adapt to downstream tasks
effectively with only a few parameters, while also improving the model's
generalization ability on the downstream tasks. Extensive experiments show that
AdapterGNN achieves higher evaluation performance (outperforming full
fine-tuning by 1.4% and 5.5% in the chemistry and biology domains respectively,
with only 5% of its parameters tuned) and lower generalization gaps compared to
full fine-tuning. Moreover, we empirically show that a larger GNN model can
have a worse generalization ability, which differs from the trend observed in
large language models. We have also provided a theoretical justification for
delta tuning can improve the generalization ability of GNNs by applying
generalization bounds.

本研究提出了一种新的 delta 调节方法（AdapterGNN），用于图神经网络，通过具有高度表现力的适配器实现对下游任务的适应，从而仅使用少量参数，在化学和生物领域分别超过了完全微调 1.4％和 5.5％。实验结果表明 AdapterGNN 达到了更高的评估性能，并在广义化程度方面比完全微调表现更好。