Emerging multi-model workloads with heavy models like recent large language
models significantly increased the compute and memory demands on hardware. To
address such increasing demands, designing a scalable hardware architecture
became a key problem. Among recent solutions, the 2.5D silicon interposer
multi-chip module (MCM)-based AI accelerator has been actively explored as a
promising scalable solution due to their significant benefits in the low
engineering cost and composability. However, previous MCM accelerators are
based on homogeneous architectures with fixed dataflow, which encounter major
challenges from highly heterogeneous multi-model workloads due to their limited
workload adaptivity. Therefore, in this work, we explore the opportunity in the
heterogeneous dataflow MCM AI accelerators. We identify the scheduling of
multi-model workload on heterogeneous dataflow MCM AI accelerator is an
important and challenging problem due to its significance and scale, which
reaches O(10^18) scale even for a single model case on 6x6 chiplets. We develop
a set of heuristics to navigate the huge scheduling space and codify them into
a scheduler with advanced techniques such as inter-chiplet pipelining. Our
evaluation on ten multi-model workload scenarios for datacenter multitenancy
and AR/VR use-cases has shown the efficacy of our approach, achieving on
average 35.3% and 31.4% less energy-delay product (EDP) for the respective
applications settings compared to homogeneous baselines.

最近的大规模语言模型等新型大规模混合模态工作负载大大增加了硬件的计算和内存需求。为了应对不断增长的需求，设计可扩展的硬件架构成为一个关键问题。在最近的解决方案中，基于 2.5D 硅互联器多芯片模块（MCM）的人工智能加速器已被广泛探索，由于其在低工程成本和可组合性方面具有显著优势。然而，以前的 MCM 加速器是基于具有固定数据流的同构架构，对于高度异构的多模型工作负载存在重大挑战，因为它们的工作负载适应性有限。因此，在本研究中，我们探索了采用异构数据流 MCM 人工智能加速器的机会。我们确定了在异构数据流 MCM 人工智能加速器上进行多模型工作负载调度是一个重要且具有挑战性的问题，由于其重要性和规模，即使在 6x6 芯片的单模型情况下，也达到 O（10 ^ 18）规模。我们开发了一组启发式方法来遍历巨大的调度空间，并将它们整理成具有高级技术（如芯片间流水线）的调度器。我们对十种多模型工作负载场景进行的评估（包括数据中心多租户和增强现实 / 虚拟现实应用）显示了我们方法的有效性，相比于同构基线，平均能减少 35.3% 和 31.4% 的能量 - 延迟产品（EDP）。