Text-rich graphs, prevalent in data mining contexts like e-commerce and
academic graphs, consist of nodes with textual features linked by various
relations. Traditional graph machine learning models, such as Graph Neural
Networks (GNNs), excel in encoding the graph structural information, but have
limited capability in handling rich text on graph nodes. Large Language Models
(LLMs), noted for their superior text understanding abilities, offer a solution
for processing the text in graphs but face integration challenges due to their
limitation for encoding graph structures and their computational complexities
when dealing with extensive text in large neighborhoods of interconnected
nodes. This paper introduces ``Hierarchical Compression'' (HiCom), a novel
method to align the capabilities of LLMs with the structure of text-rich
graphs. HiCom processes text in a node's neighborhood in a structured manner by
organizing the extensive textual information into a more manageable hierarchy
and compressing node text step by step. Therefore, HiCom not only preserves the
contextual richness of the text but also addresses the computational challenges
of LLMs, which presents an advancement in integrating the text processing power
of LLMs with the structural complexities of text-rich graphs. Empirical results
show that HiCom can outperform both GNNs and LLM backbones for node
classification on e-commerce and citation graphs. HiCom is especially effective
for nodes from a dense region in a graph, where it achieves a 3.48% average
performance improvement on five datasets while being more efficient than LLM
backbones.

在处理文本丰富的图结构数据中，本研究介绍了一种被称为 “Hierarchical Compression” 的新方法，通过将文本信息组织成可管理的层次结构并逐步压缩节点文本，使得大型语言模型能够更好地与文本丰富的图结构进行整合，不仅能保留文本的上下文信息，还能解决计算复杂性问题，并在电子商务和引用图上实现了比传统方法（如图神经网络和大型语言模型）更好的节点分类性能。

基于大型语言模型的文本丰富图的分层压缩

Hierarchical Compression of Text-Rich Graphs via Large Language Models

Text-rich graphs, which exhibit rich textual information on nodes and edges,
are prevalent across a wide range of real-world business applications. Large
Language Models (LLMs) have demonstrated remarkable abilities in understanding
text, which also introduced the potential for more expressive modeling in
text-rich graphs. Despite these capabilities, efficiently applying LLMs to
representation learning on graphs presents significant challenges. Recently,
parameter-efficient fine-tuning methods for LLMs have enabled efficient new
task generalization with minimal time and memory consumption. Inspired by this,
we introduce Graph-aware Parameter-Efficient Fine-Tuning - GPEFT, a novel
approach for efficient graph representation learning with LLMs on text-rich
graphs. Specifically, we utilize a graph neural network (GNN) to encode
structural information from neighboring nodes into a graph prompt. This prompt
is then inserted at the beginning of the text sequence. To improve the quality
of graph prompts, we pre-trained the GNN to assist the frozen LLM in predicting
the next token in the node text. Compared with existing joint GNN and LMs, our
method directly generate the node embeddings from large language models with an
affordable fine-tuning cost. We validate our approach through comprehensive
experiments conducted on 8 different text-rich graphs, observing an average
improvement of 2% in hit@1 and Mean Reciprocal Rank (MRR) in link prediction
evaluations. Our results demonstrate the efficacy and efficiency of our model,
showing that it can be smoothly integrated with various large language models,
including OPT, LLaMA and Falcon.

针对文本丰富的图表，我们引入了一种新方法 —— 图感知参数高效微调（GPEFT），通过利用图神经网络（GNN）编码邻居节点的结构信息生成图提示，插入到文本序列的开头，从而实现了高效的图表示学习。通过在 8 个不同的文本丰富的图表上进行全面实验证明了该方法的有效性和高效性，在链接预测评估中平均提升了 2% 的 hit@1 和平均倒数排名（MRR），同时表明可以与多种大型语言模型无缝集成，包括 OPT、LLaMA 和 Falcon。