Graph Neural Networks (GNNs) have shown promising performance in various
graph learning tasks, but at the cost of resource-intensive computations. The
primary overhead of GNN update stems from graph propagation and weight
transformation, both involving operations on graph-scale matrices. Previous
studies attempt to reduce the computational budget by leveraging graph-level or
network-level sparsification techniques, resulting in downsized graph or
weights. In this work, we propose Unifews, which unifies the two operations in
an entry-wise manner considering individual matrix elements, and conducts joint
edge-weight sparsification to enhance learning efficiency. The entry-wise
design of Unifews enables adaptive compression across GNN layers with
progressively increased sparsity, and is applicable to a variety of
architectural designs with on-the-fly operation simplification. Theoretically,
we establish a novel framework to characterize sparsified GNN learning in view
of a graph optimization process, and prove that Unifews effectively
approximates the learning objective with bounded error and reduced
computational load. We conduct extensive experiments to evaluate the
performance of our method in diverse settings. Unifews is advantageous in
jointly removing more than 90% of edges and weight entries with comparable or
better accuracy than baseline models. The sparsification offers remarkable
efficiency improvements including 10-20x matrix operation reduction and up to
100x acceleration in graph propagation time for the largest graph at the
billion-edge scale.

通过在个别矩阵元素上考虑统一图传播和权重变换操作，并进行联合边权稀疏化以提高学习效率，Unifews 在逐渐增加稀疏性的情况下适用于不同的架构设计，并在理论上证明了较低的计算负载以及稳定的近似学习目标。通过广泛的实验评估，Unifews 在移除超过 90% 的边和权重条目的同时，与基准模型相当或更高的准确性方面具有优势，并在效率方面提供了显着的改进。

Unifews：用于图神经网络高效稀疏化的统一逐元素稀疏化

Unifews: Unified Entry-Wise Sparsification for Efficient Graph Neural  Network

A pivotal advancement in the progress of large language models (LLMs) is the
emergence of the Mixture-of-Experts (MoE) LLMs. Compared to traditional LLMs,
MoE LLMs can achieve higher performance with fewer parameters, but it is still
hard to deploy them due to their immense parameter sizes. Different from
previous weight pruning methods that rely on specifically designed hardware,
this paper mainly aims to enhance the deployment efficiency of MoE LLMs by
introducing plug-and-play expert-level sparsification techniques. Specifically,
we propose, for the first time to our best knowledge, post-training approaches
for task-agnostic and task-specific expert pruning and skipping of MoE LLMs,
tailored to improve deployment efficiency while maintaining model performance
across a wide range of tasks. Extensive experiments show that our proposed
methods can simultaneously reduce model sizes and increase the inference speed,
while maintaining satisfactory performance. Data and code will be available at
this https URL

通过引入插拔式专家级稀疏化技术，本文主要针对 MoE LLMs 的部署效率进行了改进，提出了任务无关和任务特定的专家修剪和跳过方法，从而同时减小模型大小、增加推理速度并保持满意的性能。