Large Language Models (LLMs) have shown promising results on various language
and vision tasks. Recently, there has been growing interest in applying LLMs to
graph-based tasks, particularly on Text-Attributed Graphs (TAGs). However, most
studies have focused on node classification, while the use of LLMs for link
prediction (LP) remains understudied. In this work, we propose a new task on
LLMs, where the objective is to leverage LLMs to predict missing links between
nodes in a graph. This task evaluates an LLM's ability to reason over
structured data and infer new facts based on learned patterns. This new task
poses two key challenges: (1) How to effectively integrate pairwise structural
information into the LLMs, which is known to be crucial for LP performance, and
(2) how to solve the computational bottleneck when teaching LLMs to perform LP.
To address these challenges, we propose LinkGPT, the first end-to-end trained
LLM for LP tasks. To effectively enhance the LLM's ability to understand the
underlying structure, we design a two-stage instruction tuning approach where
the first stage fine-tunes the pairwise encoder, projector, and node projector,
and the second stage further fine-tunes the LLMs to predict links. To address
the efficiency challenges at inference time, we introduce a retrieval-reranking
scheme. Experiments show that LinkGPT can achieve state-of-the-art performance
on real-world graphs as well as superior generalization in zero-shot and
few-shot learning, surpassing existing benchmarks. At inference time, it can
achieve $10\times$ speedup while maintaining high LP accuracy.

本研究提出了 LinkGPT，首个用于链接预测任务的端到端训练的大型语言模型 (LLM)，通过有效整合结构信息和解决效率问题，在真实世界的图和零样本学习中取得了领先水平的性能。

LinkGPT: 教授大型语言模型预测丢失链接

LinkGPT: Teaching Large Language Models To Predict Missing Links

Transformer-based long context generative models power emerging AI
applications like hour-long video understanding and project-level coding agent.
Deploying long context transformers (e.g., 100K to 10M tokens) is prohibitively
expensive compared to short context (e.g., 4K tokens) model variants. Reducing
the cost of long-context transformers is becoming a pressing research and
engineering challenge starting from the year of 2024. This work describes a
concurrent programming framework for quantitatively analyzing the efficiency
challenges in serving multiple long-context requests under limited size of GPU
high-bandwidth memory (HBM) regime. We give a detailed analysis of how all
additional computational costs, compared to 4K context, trace back to
\textit{one single source: the large size of the KV cache}. We use a 34B
GPT-3.5 level model of 50K context on A100 NVLink as a running example, and
describe how its large KV cache causes four types of deployment challenges: (1)
prefilling long inputs takes much longer compute time and GPU memory than short
inputs; (2) after prefilling, the large KV cache residing on the GPU HBM
substantially restricts the number of concurrent users being served; (3) during
decoding, repeatedly reading the KV cache from HBM to SM largely increases
latency; (4) when KV cache memory overflows, swapping it from HBM to DDR causes
significant context switching latency. We use this framework to analyze
existing works and identify possibilities of combining them to build end-to-end
systems. Overall, this work offers a foundational framework for analyzing long
context transformer deployment and identifies directions towards reducing the
inference cost of 1M context to be as cheap as 4K.

为了降低长上下文 Transformer 模型的成本并解决效率挑战，本研究提出了一种并行编程框架，用于定量分析在 GPU 高带宽内存限制下为多个长上下文请求提供服务时所面临的效率挑战，并识别出减少 1M 上下文推断成本的可能方向。

部署长上下文变压器的挑战：理论峰值性能分析

Challenges in Deploying Long-Context Transformers: A Theoretical Peak  Performance Analysis

Understanding and explaining the predictions of Graph Neural Networks (GNNs),
is crucial for enhancing their safety and trustworthiness. Subgraph-level
explanations are gaining attention for their intuitive appeal. However, most
existing subgraph-level explainers face efficiency challenges in explaining
GNNs due to complex search processes. The key challenge is to find a balance
between intuitiveness and efficiency while ensuring transparency. Additionally,
these explainers usually induce subgraphs by nodes, which may introduce
less-intuitive disconnected nodes in the subgraph-level explanations or omit
many important subgraph structures. In this paper, we reveal that inducing
subgraph explanations by edges is more comprehensive than other subgraph
inducing techniques. We also emphasize the need of determining the subgraph
explanation size for each data instance, as different data instances may
involve different important substructures. Building upon these considerations,
we introduce a training-free approach, named EiG-Search. We employ an efficient
linear-time search algorithm over the edge-induced subgraphs, where the edges
are ranked by an enhanced gradient-based importance. We conduct extensive
experiments on a total of seven datasets, demonstrating its superior
performance and efficiency both quantitatively and qualitatively over the
leading baselines.

图神经网络的预测解释对于增强其安全性和可信度至关重要。本文介绍了一种名为 EiG-Search 的无需训练的方法，通过边诱导的子图进行高效线性时间搜索，边的重要性由增强的基于梯度的方法进行排序，实验结果在七个数据集上显示了其在性能和效率上的优越性。