Rich textual and topological information of textual graphs need to be modeled
in real-world applications such as webpages, e-commerce, and academic articles.
Practitioners have been long following the path of adopting a shallow text
encoder and a subsequent graph neural network (GNN) to solve this problem. In
light of recent advancements in large language models (LLMs), it is apparent
that integrating LLMs for enhanced textual encoding can substantially improve
the performance of textual graphs. Nevertheless, the efficiency of these
methods poses a significant challenge. In this paper, we propose ENGINE, a
parameter- and memory-efficient fine-tuning method for textual graphs with an
LLM encoder. The key insight is to combine the LLMs and GNNs through a tunable
side structure, which significantly reduces the training complexity without
impairing the joint model's capacity. Extensive experiments on textual graphs
demonstrate our method's effectiveness by achieving the best model performance,
meanwhile having the lowest training cost compared to previous methods.
Moreover, we introduce two variants with caching and dynamic early exit to
further enhance training and inference speed. Specifically, caching accelerates
ENGINE's training by 12x, and dynamic early exit achieves up to 5x faster
inference with a negligible performance drop (at maximum 1.17% relevant drop
across 7 datasets).

通过将大型语言模型（LLMs）与图神经网络（GNNs）相结合，我们提出了一个参数和内存效率的 FINE-TUNING 方法来处理文本图，该方法通过可调的边结构显著降低了训练复杂性，同时在文本图上获得了最佳模型性能，且训练成本最低。

基于文本图的大型语言模型的高效调优和推理

Efficient Tuning and Inference for Large Language Models on Textual  Graphs

Textual graphs (TGs) are graphs whose nodes correspond to text (sentences or
documents), which are widely prevalent. The representation learning of TGs
involves two stages: (i) unsupervised feature extraction and (ii) supervised
graph representation learning. In recent years, extensive efforts have been
devoted to the latter stage, where Graph Neural Networks (GNNs) have dominated.
However, the former stage for most existing graph benchmarks still relies on
traditional feature engineering techniques. More recently, with the rapid
development of language models (LMs), researchers have focused on leveraging
LMs to facilitate the learning of TGs, either by jointly training them in a
computationally intensive framework (merging the two stages), or designing
complex self-supervised training tasks for feature extraction (enhancing the
first stage). In this work, we present SimTeG, a frustratingly Simple approach
for Textual Graph learning that does not innovate in frameworks, models, and
tasks. Instead, we first perform supervised parameter-efficient fine-tuning
(PEFT) on a pre-trained LM on the downstream task, such as node classification.
We then generate node embeddings using the last hidden states of finetuned LM.
These derived features can be further utilized by any GNN for training on the
same task. We evaluate our approach on two fundamental graph representation
learning tasks: node classification and link prediction. Through extensive
experiments, we show that our approach significantly improves the performance
of various GNNs on multiple graph benchmarks.

利用简化且无创新的方法在下游任务上进行预训练语言模型的有监督参数高效微调，并生成节点嵌入以提高图形神经网络在多个图形基准上的性能。

SimTeG：一个令人沮丧却简单的方法，提升文本图学习能力

SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning

How can we learn effective node representations on textual graphs? Graph
Neural Networks (GNNs) that use Language Models (LMs) to encode textual
information of graphs achieve state-of-the-art performance in many node
classification tasks. Yet, combining GNNs with LMs has not been widely explored
for practical deployments due to its scalability issues. In this work, we
tackle this challenge by developing a Graph-Aware Distillation framework (GRAD)
to encode graph structures into an LM for graph-free, fast inference. Different
from conventional knowledge distillation, GRAD jointly optimizes a GNN teacher
and a graph-free student over the graph's nodes via a shared LM. This
encourages the graph-free student to exploit graph information encoded by the
GNN teacher while at the same time, enables the GNN teacher to better leverage
textual information from unlabeled nodes. As a result, the teacher and the
student models learn from each other to improve their overall performance.
Experiments in eight node classification benchmarks in both transductive and
inductive settings showcase GRAD's superiority over existing distillation
approaches for textual graphs.

本论文针对在文本图上学习有效节点表示的问题，提出了一种使用语言模型编码图文本信息的图神经网络和知识蒸馏机制，能够在无图情况下实现快速推理，实验结果表明，该方法在节点分类任务上具有显著优势。