Machine Learning (ML) operators are the building blocks to design ML models
with various target applications. GEneral Matrix Multiplication (GEMM)
operators are the backbone of ML models. They are notorious for being
computationally expensive requiring billions of multiply-and-accumulate.
Therefore, significant effort has been put to study and optimize the GEMM
operators in order to speed up the execution of ML models. GPUs and
accelerators are widely deployed to accelerate ML workloads by optimizing the
execution of GEMM operators. Nonetheless, the performance of NonGEMM operators
have not been studied as thoroughly as GEMMs. Therefore, this paper describes
\bench, a benchmark to study NonGEMM operators. We first construct \bench using
popular ML workloads from different domains, then perform case studies on
various grade GPU platforms to analyze the behavior of NonGEMM operators in GPU
accelerated systems. Finally, we present some key takeaways to bridge the gap
between GEMM and NonGEMM operators and to offer the community with potential
new optimization directions.

机器学习 (ML) 运算符是设计具有不同目标应用的 ML 模型的基本构建模块。GEneral Matrix Multiplication (GEMM) 运算符是 ML 模型的基础，但在计算上非常昂贵，因此需要对其进行研究和优化以加快 ML 模型的执行速度。然而，非 GEMM 运算符的性能尚未像 GEMMs 那样进行深入研究，因此本文介绍了一种用于研究非 GEMM 运算符的基准测试，通过使用来自不同领域的流行 ML 工作负载构建 ench，然后在各个等级的 GPU 平台上进行案例研究，以分析 GPU 加速系统中非 GEMM 运算符的行为，并提供一些关键性的发现，以弥合 GEMM 和非 GEMM 运算符之间的差距，并为社区提供潜在的新的优化方向。

非 GEMM 基准：通过非 GEMM 工作负载了解最新机器学习工作负载的性能边界

NonGEMM Bench: Understanding the Performance Horizon of the Latest ML  Workloads with NonGEMM Workloads

In this study, we introduce NeuralMatrix, a novel framework that enables the
computation of versatile deep neural networks (DNNs) on a single general matrix
multiplication (GEMM) accelerator. The proposed approach overcomes the
specificity limitations of ASIC-based accelerators while achieving
application-specific acceleration levels compared to general-purpose processors
such as CPUs and GPUs. We address the challenges of mapping both linear and
nonlinear operations in DNN computation to general matrix multiplications and
the impact of using a GEMM accelerator on DNN inference accuracy. Extensive
experiments are conducted on various DNN models from three popular categories
(i.e., CNN, Transformers, and GNN) as illustrative backbone models. Our results
demonstrate that DNNs suffer only up to a 2.02% accuracy loss after being
converted to general matrix multiplication, while achieving 113x to 19.44x
improvements in throughput per power compared to CPUs and GPUs.

本研究提出了一种新的框架 NeuralMatrix，可以在一个单一的通用矩阵乘法加速器上计算多功能的深度神经网络，同时在性能和特定应用的加速水平方面与 CPU 和 GPU 等通用处理器相比具有优势。

NeuralMatrix：将整个神经网络转换为通用矩阵乘法以实现高效推断

NeuralMatrix: Moving Entire Neural Networks to General Matrix  Multiplication for Efficient Inference

Deep neural networks (DNNs) require very large amounts of computation both
for training and for inference when deployed in the field. A common approach to
implementing DNNs is to recast the most computationally expensive operations as
general matrix multiplication (GEMM). However, as we demonstrate in this paper,
there are a great many different ways to express DNN convolution operations
using GEMM. Although different approaches all perform the same number of
operations, the size of temporary data structures differs significantly.
Convolution of an input matrix with dimensions $C \times H \times W$, requires
$O(K^2CHW)$ additional space using the classical im2col approach. More recently
memory-efficient approaches requiring just $O(KCHW)$ auxiliary space have been
proposed.
We present two novel GEMM-based algorithms that require just $O(MHW)$ and
$O(KW)$ additional space respectively, where $M$ is the number of channels in
the result of the convolution. These algorithms dramatically reduce the space
overhead of DNN convolution, making it much more suitable for memory-limited
embedded systems. Experimental evaluation shows that our low-memory algorithms
are just as fast as the best patch-building approaches despite requiring just a
fraction of the amount of additional memory. Our low-memory algorithms have
excellent data locality which gives them a further edge over patch-building
algorithms when multiple cores are used. As a result, our low memory algorithms
often outperform the best patch-building algorithms using multiple threads.

本文提出两种新型基于 GEMM 的算法，分别只需要额外的 O (MHW) 和 O (KW) 的空间，显著降低了 DNN 卷积的空间开销，适用于内存受限的嵌入式系统，并且实验表明我们的低内存算法和最好的图案构建方法一样快，尽管需要的额外内存只相当于后者的一小部分。