Pretraining plays a pivotal role in acquiring generalized knowledge from
large-scale data, achieving remarkable successes as evidenced by large models
in CV and NLP. However, progress in the graph domain remains limited due to
fundamental challenges such as feature heterogeneity and structural
heterogeneity. Recently, increasing efforts have been made to enhance node
feature quality with Large Language Models (LLMs) on text-attributed graphs
(TAGs), demonstrating superiority to traditional bag-of-words or word2vec
techniques. These high-quality node features reduce the previously critical
role of graph structure, resulting in a modest performance gap between Graph
Neural Networks (GNNs) and structure-agnostic Multi-Layer Perceptrons (MLPs).
Motivated by this, we introduce a feature-centric pretraining perspective by
treating graph structure as a prior and leveraging the rich, unified feature
space to learn refined interaction patterns that generalizes across graphs. Our
framework, Graph Sequence Pretraining with Transformer (GSPT), samples node
contexts through random walks and employs masked feature reconstruction to
capture pairwise proximity in the LLM-unified feature space using a standard
Transformer. By utilizing unified text representations rather than varying
structures, our framework achieves significantly better transferability among
graphs within the same domain. GSPT can be easily adapted to both node
classification and link prediction, demonstrating promising empirical success
on various datasets.

图序列预训练框架 GSPT 利用统一的文本表示，在图领域中取得了显著的可转移性和实证成功。