Graph contrastive learning (GCL) has become a powerful tool for learning
graph data, but its scalability remains a significant challenge. In this work,
we propose a simple yet effective training framework called Structural
Compression (StructComp) to address this issue. Inspired by a sparse low-rank
approximation on the diffusion matrix, StructComp trains the encoder with the
compressed nodes. This allows the encoder not to perform any message passing
during the training stage, and significantly reduces the number of sample pairs
in the contrastive loss. We theoretically prove that the original GCL loss can
be approximated with the contrastive loss computed by StructComp. Moreover,
StructComp can be regarded as an additional regularization term for GCL models,
resulting in a more robust encoder. Empirical studies on seven benchmark
datasets show that StructComp greatly reduces the time and memory consumption
while improving model performance compared to the vanilla GCL models and
scalable training methods.

通过稀疏低秩近似方法，提出了一种名为 StructComp 的简单而有效的训练框架，用于解决图形对比学习的可扩展性问题，并在理论上证明了 StructComp 可以近似原始的对比损失，并改进了模型性能。

StructComp: 在图对比学习中用结构压缩代替传播的训练

StructComp: Substituting propagation with Structural Compression in  Training Graph Contrastive Learning

In this paper, we consider the problem of improving the inference latency of
language model-based dense retrieval systems by introducing structural
compression and model size asymmetry between the context and query encoders.
First, we investigate the impact of pre and post-training compression on the
MSMARCO, Natural Questions, TriviaQA, SQUAD, and SCIFACT, finding that
asymmetry in the dual encoders in dense retrieval can lead to improved
inference efficiency. Knowing this, we introduce Kullback Leibler Alignment of
Embeddings (KALE), an efficient and accurate method for increasing the
inference efficiency of dense retrieval methods by pruning and aligning the
query encoder after training. Specifically, KALE extends traditional Knowledge
Distillation after bi-encoder training, allowing for effective query encoder
compression without full retraining or index generation. Using KALE and
asymmetric training, we can generate models which exceed the performance of
DistilBERT despite having 3x faster inference.

本文讨论了引入结构压缩和编码器大小不对称性来改善基于语言模型的密集检索系统的推理延迟问题，并引入了 Kullback Leibler 嵌入对齐（KALE）方法以提高密集检索方法的推理效率。使用 KALE 和不对称训练，我们可以生成优于 DistilBERT 且推理速度快 3 倍的模型。