Limited by the complexity of basis function (B-spline) calculations,
Kolmogorov-Arnold Networks (KAN) suffer from restricted parallel computing
capability on GPUs. This paper proposes a novel ReLU-KAN implementation that
inherits the core idea of KAN. By adopting ReLU (Rectified Linear Unit) and
point-wise multiplication, we simplify the design of KAN's basis function and
optimize the computation process for efficient CUDA computing. The proposed
ReLU-KAN architecture can be readily implemented on existing deep learning
frameworks (e.g., PyTorch) for both inference and training. Experimental
results demonstrate that ReLU-KAN achieves a 20x speedup compared to
traditional KAN with 4-layer networks. Furthermore, ReLU-KAN exhibits a more
stable training process with superior fitting ability while preserving the
"catastrophic forgetting avoidance" property of KAN. You can get the code in
this https URL

该论文介绍了一种新颖的 ReLU-KAN 实现，通过采用 ReLU 和逐元素乘法简化了 KAN 的基函数设计，并优化了计算过程，提高了 CUDA 计算的效率。实验结果表明，与传统的 KAN 相比，ReLU-KAN 在具有 4 层网络的情况下加速了 20 倍。此外，ReLU-KAN 在保留了 KAN 的 “防止灾难性遗忘” 特性的同时，表现出更稳定的训练过程和较好的拟合能力。

ReLU-KAN: 仅需矩阵加法、点乘和 ReLU 的新型 Kolmogorov-Arnold 网络

ReLU-KAN: New Kolmogorov-Arnold Networks that Only Need Matrix Addition,  Dot Multiplication, and ReLU

Deep learning models can exhibit what appears to be a sudden ability to solve
a new problem as training time ($T$), training data ($D$), or model size ($N$)
increases, a phenomenon known as emergence. In this paper, we present a
framework where each new ability (a skill) is represented as a basis function.
We solve a simple multi-linear model in this skill-basis, finding analytic
expressions for the emergence of new skills, as well as for scaling laws of the
loss with training time, data size, model size, and optimal compute ($C$). We
compare our detailed calculations to direct simulations of a two-layer neural
network trained on multitask sparse parity, where the tasks in the dataset are
distributed according to a power-law. Our simple model captures, using a single
fit parameter, the sigmoidal emergence of multiple new skills as training time,
data size or model size increases in the neural network.

深度学习模型的发展速度、数据量和模型规模的增加会导致其展示出解决新问题的突然能力，这被称为发生现象。本文提出了一个基于技能的框架，其中每一项新能力都被表示为基函数。我们在这个技能基上解决了一个简单的多线性模型，并得到了与培训时间、数据量、模型规模和最佳计算（C）有关的新能力的出现以及丢失的尺度规律的解析表达式。我们将详细的计算结果与在多任务稀疏奇偶性训练数据上训练的两层神经网络的直接模拟进行了比较，其中数据集的任务分布符合幂律分布。我们的简单模型使用了单一的拟合参数，能够捕捉到神经网络中随着训练时间、数据量或模型规模增加而出现的多个新技能的 S 型发生现象。