Text-rich graphs, which exhibit rich textual information on nodes and edges,
are prevalent across a wide range of real-world business applications. Large
Language Models (LLMs) have demonstrated remarkable abilities in understanding
text, which also introduced the potential for more expressive modeling in
text-rich graphs. Despite these capabilities, efficiently applying LLMs to
representation learning on graphs presents significant challenges. Recently,
parameter-efficient fine-tuning methods for LLMs have enabled efficient new
task generalization with minimal time and memory consumption. Inspired by this,
we introduce Graph-aware Parameter-Efficient Fine-Tuning - GPEFT, a novel
approach for efficient graph representation learning with LLMs on text-rich
graphs. Specifically, we utilize a graph neural network (GNN) to encode
structural information from neighboring nodes into a graph prompt. This prompt
is then inserted at the beginning of the text sequence. To improve the quality
of graph prompts, we pre-trained the GNN to assist the frozen LLM in predicting
the next token in the node text. Compared with existing joint GNN and LMs, our
method directly generate the node embeddings from large language models with an
affordable fine-tuning cost. We validate our approach through comprehensive
experiments conducted on 8 different text-rich graphs, observing an average
improvement of 2% in hit@1 and Mean Reciprocal Rank (MRR) in link prediction
evaluations. Our results demonstrate the efficacy and efficiency of our model,
showing that it can be smoothly integrated with various large language models,
including OPT, LLaMA and Falcon.

针对文本丰富的图表，我们引入了一种新方法 —— 图感知参数高效微调（GPEFT），通过利用图神经网络（GNN）编码邻居节点的结构信息生成图提示，插入到文本序列的开头，从而实现了高效的图表示学习。通过在 8 个不同的文本丰富的图表上进行全面实验证明了该方法的有效性和高效性，在链接预测评估中平均提升了 2% 的 hit@1 和平均倒数排名（MRR），同时表明可以与多种大型语言模型无缝集成，包括 OPT、LLaMA 和 Falcon。