Recent research in deep learning (DL) has investigated the use of the Fast
Fourier Transform (FFT) to accelerate the computations involved in
Convolutional Neural Networks (CNNs) by replacing spatial convolution with
element-wise multiplications on the spectral domain. These approaches mainly
rely on the FFT to reduce the number of operations, which can be further
decreased by adopting the Real-Valued FFT. In this paper, we propose using the
phasor form, a polar representation of complex numbers, as a more efficient
alternative to the traditional approach. The experimental results, evaluated on
the CIFAR-10, demonstrate that our method achieves superior speed improvements
of up to a factor of 1.376 (average of 1.316) during training and up to 1.390
(average of 1.321) during inference when compared to the traditional
rectangular form employed in modern CNN architectures. Similarly, when
evaluated on the CIFAR-100, our method achieves superior speed improvements of
up to a factor of 1.375 (average of 1.299) during training and up to 1.387
(average of 1.300) during inference. Most importantly, given the modular aspect
of our approach, the proposed method can be applied to any existing
convolution-based DL model without design changes.

通过使用极相表达形式作为替代传统方法的更高效的选择，我们的方法在 CIFAR-10 和 CIFAR-100 数据集上相较于现代 CNN 架构采用的矩形形式，实现了训练和推断速度的最高提升分别达到 1.376 和 1.390。同时，我们的方法可应用于任何基于卷积的深度学习模型，而无需进行设计更改。

基于相量驱动的快速傅里叶变换卷积神经网络加速

Phasor-Driven Acceleration for FFT-based CNNs

We present an approximate attention mechanism named HyperAttention to address
the computational challenges posed by the growing complexity of long contexts
used in Large Language Models (LLMs). Recent work suggests that in the
worst-case scenario, quadratic time is necessary unless the entries of the
attention matrix are bounded or the matrix has low stable rank. We introduce
two parameters which measure: (1) the max column norm in the normalized
attention matrix, and (2) the ratio of row norms in the unnormalized attention
matrix after detecting and removing large entries. We use these fine-grained
parameters to capture the hardness of the problem. Despite previous lower
bounds, we are able to achieve a linear time sampling algorithm even when the
matrix has unbounded entries or a large stable rank, provided the above
parameters are small. HyperAttention features a modular design that easily
accommodates integration of other fast low-level implementations, particularly
FlashAttention. Empirically, employing Locality Sensitive Hashing (LSH) to
identify large entries, HyperAttention outperforms existing methods, giving
significant speed improvements compared to state-of-the-art solutions like
FlashAttention. We validate the empirical performance of HyperAttention on a
variety of different long-context length datasets. For example, HyperAttention
makes the inference time of ChatGLM2 50\% faster on 32k context length while
perplexity increases from 5.6 to 6.3. On larger context length, e.g., 131k,
with causal masking, HyperAttention offers 5-fold speedup on a single attention
layer.

我们提出了一个名为 HyperAttention 的近似注意力机制，用于解决大型语言模型（LLMs）中使用的复杂长上下文所带来的计算挑战。通过引入两个参数来衡量问题的难度，我们能够实现线性时间采样算法，即使矩阵具有无界的条目或较大的稳定秩。HyperAttention 具有模块化设计，容易集成其他快速底层实现，特别是 FlashAttention。实证上，通过利用局部敏感哈希（LSH）来识别大条目，HyperAttention 优于现有方法，在与 FlashAttention 等最先进解决方案相比，获得了显著的速度改进。我们验证了 HyperAttention 在不同长上下文数据集上的实证性能，例如，ChatGLM2 的推理时间得以加快 50％，而困惑度从 5.6 增加到 6.3。对于更大的上下文长度，例如 131k，并采取因果屏蔽，HyperAttention 在单个注意层上实现了 5 倍的加速。