Pre-training GNNs to extract transferable knowledge and apply it to
downstream tasks has become the de facto standard of graph representation
learning. Recent works focused on designing self-supervised pre-training tasks
to extract useful and universal transferable knowledge from large-scale
unlabeled data. However, they have to face an inevitable question: traditional
pre-training strategies that aim at extracting useful information about
pre-training tasks, may not extract all useful information about the downstream
task. In this paper, we reexamine the pre-training process within traditional
pre-training and fine-tuning frameworks from the perspective of Information
Bottleneck (IB) and confirm that the forgetting phenomenon in pre-training
phase may cause detrimental effects on downstream tasks. Therefore, we propose
a novel \underline{D}elayed \underline{B}ottlenecking \underline{P}re-training
(DBP) framework which maintains as much as possible mutual information between
latent representations and training data during pre-training phase by
suppressing the compression operation and delays the compression operation to
fine-tuning phase to make sure the compression can be guided with labeled
fine-tuning data and downstream tasks. To achieve this, we design two
information control objectives that can be directly optimized and further
integrate them into the actual model design. Extensive experiments on both
chemistry and biology domains demonstrate the effectiveness of DBP.

传统的预训练和微调流程中的遗忘现象可能对下游任务产生不利影响，因此我们提出了一种新颖的延迟瓶颈预训练（DBP）框架，通过抑制压缩操作并延迟至微调阶段来尽量保持潜在表示与训练数据之间的互信息，以确保压缩能够由有标签的微调数据和下游任务进行引导。