Recent developments in Language Models (LMs) have shown their effectiveness
in NLP tasks, particularly in knowledge-intensive tasks. However, the
mechanisms underlying knowledge storage and memory access within their
parameters remain elusive. In this paper, we investigate whether a generative
LM (e.g., GPT-2) is able to access its memory sequentially or randomly. Through
carefully-designed synthetic tasks, covering the scenarios of full recitation,
selective recitation and grounded question answering, we reveal that LMs manage
to sequentially access their memory while encountering challenges in randomly
accessing memorized content. We find that techniques including recitation and
permutation improve the random memory access capability of LMs. Furthermore, by
applying this intervention to realistic scenarios of open-domain question
answering, we validate that enhancing random access by recitation leads to
notable improvements in question answering. The code to reproduce our
experiments can be found at this https URL
com/sail-sg/lm-random-memory-access.

通过合成任务，我们研究了生成式语言模型（例如 GPT-2）能否按顺序或随机访问其记忆，并发现记忆重述和置换等技术提高了随机访问能力，进而在问答任务中取得了显着的改进。

记忆之外：语言模型中随机内存访问的挑战

Beyond Memorization: The Challenge of Random Memory Access in Language  Models

The greatest demand for today's computing is machine learning. This paper
analyzes three machine learning algorithms: transformers, spatial convolution,
and FFT. The analysis is novel in three aspects. First, it measures the cost of
memory access on an abstract memory hierarchy, instead of traditional time or
space complexity. Second, the analysis is asymptotic and identifies the primary
sources of the memory cost. Finally, the result is symbolic, which can be used
to select algorithmic parameters such as the group size in grouped query
attention for any dimension size and number of heads and the batch size for
batched convolution for any image size and kernel size.

该论文通过分析 transformers、spatial convolution 和 FFT 三种机器学习算法，提出了三个创新点。首先，它使用抽象的内存层次结构来衡量内存访问成本，而不是传统的时间或空间复杂度。其次，该分析是渐近的，并确定了内存成本的主要来源。最后，该结果是符号的，可以用于选择算法参数，例如分组查询关注力的组大小，以及任意尺寸和头数的批量卷积的批量大小。

DMC4ML: 机器学习的数据移动复杂性

DMC4ML: Data Movement Complexity for Machine Learning

Word2Vec remains one of the highly-impactful innovations in the field of
Natural Language Processing (NLP) that represents latent grammatical and
syntactical information in human text with dense vectors in a low dimension.
Word2Vec has high computational cost due to the algorithm's inherent
sequentiality, intensive memory accesses, and the large vocabularies it
represents. While prior studies have investigated technologies to explore
parallelism and improve memory system performance, they struggle to effectively
gain throughput on powerful GPUs.
We identify memory data access and latency as the primary bottleneck in prior
works on GPUs, which prevents highly optimized kernels from attaining the
architecture's peak performance. We present a novel algorithm, FULL-W2V, which
maximally exploits the opportunities for data reuse in the W2V algorithm and
leverages GPU architecture and resources to reduce access to low memory levels
and improve temporal locality. FULL-W2V is capable of reducing accesses to GPU
global memory significantly, e.g., by more than 89\%, compared to prior
state-of-the-art GPU implementations, resulting in significant performance
improvement that scales across successive hardware generations. Our prototype
implementation achieves 2.97X speedup when ported from Nvidia Pascal P100 to
Volta V100 cards, and outperforms the state-of-the-art by 5.72X on V100 cards
with the same embedding quality. In-depth analysis indicates that the reduction
of memory accesses through register and shared memory caching and
high-throughput shared memory reduction leads to a significantly improved
arithmetic intensity. FULL-W2V can potentially benefit many applications in NLP
and other domains.

通过减少内存访问和改进时序局部性，FULL-W2V 算法在 GPU 上实现了显著的性能提升，并且对 NLP 和其他领域的应用具有潜在的益处。

FULL-W2V: 在 GPU 加速系统上充分利用数据复用进行 W2V

FULL-W2V: Fully Exploiting Data Reuse for W2V on GPU-Accelerated Systems

Deep neural networks (DNNs) are of critical use in different domains. To
accelerate DNN computation, tensor compilers are proposed to generate efficient
code on different domain-specific accelerators. Existing tensor compilers
mainly focus on optimizing computation efficiency. However, memory access is
becoming a key performance bottleneck because the computational performance of
accelerators is increasing much faster than memory performance. The lack of
direct description of memory access and data dependence in current tensor
compilers' intermediate representation (IR) brings significant challenges to
generate memory-efficient code.
In this paper, we propose IntelliGen, a tensor compiler that can generate
high-performance code for memory-intensive operators by considering both
computation and data movement optimizations. IntelliGen represent a DNN program
using GIR, which includes primitives indicating its computation, data movement,
and parallel strategies. This information will be further composed as an
instruction-level dataflow graph to perform holistic optimizations by searching
different memory access patterns and computation operations, and generating
memory-efficient code on different hardware. We evaluate IntelliGen on NVIDIA
GPU, AMD GPU, and Cambricon MLU, showing speedup up to 1.97x, 2.93x, and
16.91x(1.28x, 1.23x, and 2.31x on average), respectively, compared to current
most performant frameworks.

本文提出 IntelliGen 这样一个基于 GIR 表示的张量编译器，该编译器通过综合优化不同的内存访问模式和计算操作，能够针对内存密集运算符生成高性能代码，相比当前最有效的框架，IntelliGen 在 NVIDIA GPU、AMD GPU、Cambricon MLU 上的速度提升分别达到了 1.97x、2.93x 和 16.91x。