Neighborhood attention reduces the cost of self attention by restricting each
token's attention span to its nearest neighbors. This restriction,
parameterized by a window size and dilation factor, draws a spectrum of
possible attention patterns between linear projection and self attention.
Neighborhood attention, and more generally sliding window attention patterns,
have long been bounded by infrastructure, particularly in higher-rank spaces
(2-D and 3-D), calling for the development of custom kernels, which have been
limited in either functionality, or performance, if not both. In this work, we
first show that neighborhood attention can be represented as a batched GEMM
problem, similar to standard attention, and implement it for 1-D and 2-D
neighborhood attention. These kernels on average provide 895% and 272%
improvement in full precision latency compared to existing naive kernels for
1-D and 2-D neighborhood attention respectively. We find certain inherent
inefficiencies in all unfused neighborhood attention kernels that bound their
performance and lower-precision scalability. We also developed fused
neighborhood attention; an adaptation of fused dot-product attention kernels
that allow fine-grained control over attention across different spatial axes.
Known for reducing the quadratic time complexity of self attention to a linear
complexity, neighborhood attention can now enjoy a reduced and constant memory
footprint, and record-breaking half precision latency. We observe that our
fused kernels successfully circumvent some of the unavoidable inefficiencies in
unfused implementations. While our unfused GEMM-based kernels only improve half
precision performance compared to naive kernels by an average of 496% and 113%
in 1-D and 2-D problems respectively, our fused kernels improve naive kernels
by an average of 1607% and 581% in 1-D and 2-D problems respectively.

邻域自注意力通过限制每个标记的注意力范围为其最近的邻居来降低自注意力的成本。这种限制通过窗口大小和膨胀因子参数化，绘制了在线性投影和自注意力之间的一系列可能的注意力模式。我们将邻域注意力表示为分批 GEMM 问题，实现了 1-D 和 2-D 邻域注意力，并且与现有的朴素内核相比，平均提供了 895% 和 272％的全精度延迟改进，我们观察到我们的融合内核成功地规避了未融合实现中不可避免的低效率。

更快的邻域注意力机制：在线程块级别降低自注意力的 O (n^2) 复杂度

Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self  Attention at the Threadblock Level

Time series forecasting task predicts future trends based on historical
information. Recent U-Net-based methods have demonstrated superior performance
in predicting real-world datasets. However, the performance of these models is
lower than patch-based models or linear models. In this work, we propose a
symmetric and hierarchical framework, Kernel-U-Net, which cuts the input
sequence into slices at each layer of the network and then computes them using
kernels. Furthermore, it generalizes the concept of convolutional kernels in
classic U-Net to accept custom kernels that follow the same design pattern.
Compared to the existing linear or transformer-based solution, our model
contains 3 advantages: 1) A small number of parameters: the parameters size is
$O(log(L)^2)$ where $L$ is the look-back window size, 2) Flexibility: its
kernels can be customized and fitted to the datasets, 3) Computation
efficiency: the computation complexity of transformer modules is reduced to
$O(log(L)^2)$ if they are placed close to the latent vector. Kernel-U-Net
accuracy was greater than or equal to the state-of-the-art model on six (out of
seven) real-world datasets.

时间序列预测任务基于历史信息预测未来趋势。本文提出了一种对称和分层的框架，Kernel-U-Net，它在网络的每一层将输入序列切分成片段，然后使用内核对其进行计算。与现有的线性或 Transformer 解决方案相比，我们的模型具有三个优势：1）参数较小，2）灵活性，3）计算效率。在六个（七个中的五个）真实数据集上，Kernel-U-Net 的准确性大于或等于最先进模型的准确性。