Large language models (LLMs) have recently transformed natural language
processing, enabling machines to generate human-like text and engage in
meaningful conversations. This development necessitates speed, efficiency, and
accessibility in LLM inference as the computational and memory requirements of
these systems grow exponentially. Meanwhile, advancements in computing and
memory capabilities are lagging behind, exacerbated by the discontinuation of
Moore's law. With LLMs exceeding the capacity of single GPUs, they require
complex, expert-level configurations for parallel processing. Memory accesses
become significantly more expensive than computation, posing a challenge for
efficient scaling, known as the memory wall. Here, compute-in-memory (CIM)
technologies offer a promising solution for accelerating AI inference by
directly performing analog computations in memory, potentially reducing latency
and power consumption. By closely integrating memory and compute elements, CIM
eliminates the von Neumann bottleneck, reducing data movement and improving
energy efficiency. This survey paper provides an overview and analysis of
transformer-based models, reviewing various CIM architectures and exploring how
they can address the imminent challenges of modern AI computing systems. We
discuss transformer-related operators and their hardware acceleration schemes
and highlight challenges, trends, and insights in corresponding CIM designs.

大型语言模型的推出带来了自然语言处理领域的重大变革，本文调查了与转换器模型有关的各种 CIM 构架以及它们如何解决现代人工智能计算系统面临的挑战。

记忆即一切：加速大型语言模型推理的计算于内存架构综述

Memory Is All You Need: An Overview of Compute-in-Memory Architectures  for Accelerating Large Language Model Inference

Compute-in-memory (CiM) has emerged as a compelling solution to alleviate
high data movement costs in von Neumann machines. CiM can perform massively
parallel general matrix multiplication (GEMM) operations in memory, the
dominant computation in Machine Learning (ML) inference. However, re-purposing
memory for compute poses key questions on 1) What type of CiM to use: Given a
multitude of analog and digital CiMs, determining their suitability from
systems perspective is needed. 2) When to use CiM: ML inference includes
workloads with a variety of memory and compute requirements, making it
difficult to identify when CiM is more beneficial than standard processing
cores. 3) Where to integrate CiM: Each memory level has different bandwidth and
capacity, that affects the data movement and locality benefits of CiM
integration.
In this paper, we explore answers to these questions regarding CiM
integration for ML inference acceleration. We use Timeloop-Accelergy for early
system-level evaluation of CiM prototypes, including both analog and digital
primitives. We integrate CiM into different cache memory levels in an Nvidia
A100-like baseline architecture and tailor the dataflow for various ML
workloads. Our experiments show CiM architectures improve energy efficiency,
achieving up to 0.12x lower energy than the established baseline with INT-8
precision, and upto 4x performance gains with weight interleaving and
duplication. The proposed work provides insights into what type of CiM to use,
and when and where to optimally integrate it in the cache hierarchy for GEMM
acceleration.

本文研究了在机器学习推断中将计算内存 (Compute-in-memory, CiM) 技术集成的问题，并通过实验证明了使用 CiM 架构可以提高能源效率，达到低至以 INT-8 精度 0.12 倍能源消耗和重复权重交错技术下最高 4 倍的性能增益，同时提供了关于何种类型 CiM 适合使用以及何时何地最佳集成于缓存层级进行 GEMM 加速的见解。