The increasing complexity of transformer models in artificial intelligence
expands their computational costs, memory usage, and energy consumption.
Hardware acceleration tackles the ensuing challenges by designing processors
and accelerators tailored for transformer models, supporting their computation
hotspots with high efficiency. However, memory bandwidth can hinder
improvements in hardware accelerators. Against this backdrop, in this paper we
propose a novel memory arrangement strategy, governed by the hardware
accelerator's kernel size, which effectively minimizes off-chip data access.
This arrangement is particularly beneficial for end-to-end transformer model
inference, where most of the computation is based on general matrix
multiplication (GEMM) operations. Additionally, we address the overhead of
non-GEMM operations in transformer models within the scope of this memory data
arrangement. Our study explores the implementation and effectiveness of the
proposed accelerator-driven data arrangement approach in both single- and
multi-core systems. Our evaluation demonstrates that our approach can achieve
up to a 2.8x speed increase when executing inferences employing
state-of-the-art transformers.

提出了一种新颖的内存数据排列策略，通过硬件加速器的内核大小来有效减小芯片外数据访问，特别对于基于广义矩阵乘法（GEMM）的终端到终端变压器模型推理尤为有益。本研究通过在单核和多核系统中实现和评估提出的加速器驱动的数据排列方法，证明了该方法能够实现高达 2.8 倍的速度提升。