Monolithic large language models (LLMs) like GPT-4 have paved the way for
modern generative AI applications. Training, serving, and maintaining
monolithic LLMs at scale, however, remains prohibitively expensive and
challenging. The disproportionate increase in compute-to-memory ratio of modern
AI accelerators have created a memory wall, necessitating new methods to deploy
AI. Composition of Experts (CoE) is an alternative modular approach that lowers
the cost and complexity of training and serving. However, this approach
presents two key challenges when using conventional hardware: (1) without fused
operations, smaller models have lower operational intensity, which makes high
utilization more challenging to achieve; and (2) hosting a large number of
models can be either prohibitively expensive or slow when dynamically switching
between them.
In this paper, we describe how combining CoE, streaming dataflow, and a
three-tier memory system scales the AI memory wall. We describe Samba-CoE, a
CoE system with 150 experts and a trillion total parameters. We deploy
Samba-CoE on the SambaNova SN40L Reconfigurable Dataflow Unit (RDU) - a
commercial dataflow accelerator architecture that has been co-designed for
enterprise inference and training applications. The chip introduces a new
three-tier memory system with on-chip distributed SRAM, on-package HBM, and
off-package DDR DRAM. A dedicated inter-RDU network enables scaling up and out
over multiple sockets. We demonstrate speedups ranging from 2x to 13x on
various benchmarks running on eight RDU sockets compared with an unfused
baseline. We show that for CoE inference deployments, the 8-socket RDU Node
reduces machine footprint by up to 19x, speeds up model switching time by 15x
to 31x, and achieves an overall speedup of 3.7x over a DGX H100 and 6.6x over a
DGX A100.

结合 Composition of Experts（CoE）、流数据流和三层内存系统解决了人工智能内存瓶颈的问题，并介绍了 Samba-CoE，一个包含 150 个专家和一万亿总参数的 CoE 系统，部署在 SambaNova SN40L Reconfigurable Dataflow Unit (RDU) 上，通过引入三层内存系统、专用的 RDU 网络和多个 RDU 插槽实现了速度提升，有效减少机器占用空间并加快模型切换时间。

SambaNova SN40L：通过数据流和专家组合扩展 AI 的内存墙

SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and  Composition of Experts

This paper presents ProFL, a novel progressive FL framework to effectively
break the memory wall. Specifically, ProFL divides the model into different
blocks based on its original architecture. Instead of updating the full model
in each training round, ProFL first trains the front blocks and safely freezes
them after convergence. Training of the next block is then triggered. This
process iterates until the training of the whole model is completed. In this
way, the memory footprint is effectively reduced for feasible deployment on
heterogeneous devices. In order to preserve the feature representation of each
block, we decouple the whole training process into two stages: progressive
model shrinking and progressive model growing. During the progressive model
shrinking stage, we meticulously design corresponding output modules to assist
each block in learning the expected feature representation and obtain the
initialization parameters. Then, the obtained output modules are utilized in
the corresponding progressive model growing stage. Additionally, to control the
training pace for each block, a novel metric from the scalar perspective is
proposed to assess the learning status of each block and determines when to
trigger the training of the next one. Finally, we theoretically prove the
convergence of ProFL and conduct extensive experiments on representative models
and datasets to evaluate the effectiveness of ProFL. The results demonstrate
that ProFL effectively reduces the peak memory footprint by up to 57.4% and
improves model accuracy by up to 82.4%.

通过分割模型结构，分阶段训练并冻结模型的不同部分，逐步减小模型的尺寸以降低内存占用，从而提高性能。