This paper presents MoE-Infinity, a cost-efficient mixture-of-expert (MoE)
serving system that realizes activation-aware expert offloading. MoE-Infinity
features sequence-level expert activation tracing, a new approach adept at
identifying sparse activations and capturing the temporal locality of MoE
inference. By analyzing these traces, MoE-Infinity performs novel
activation-aware expert prefetching and caching, substantially reducing the
latency overheads usually associated with offloading experts for improved cost
performance. Extensive experiments in a cluster show that MoE-Infinity
outperforms numerous existing systems and approaches, reducing latency by 4 -
20X and decreasing deployment costs by over 8X for various MoEs. MoE-Infinity's
source code is publicly available at this https URL

MoE-Infinity 是一个成本高效的专家混合系统，实现了激活感知的专家卸载。通过分析序列级专家激活追踪，MoE-Infinity 执行新颖的激活感知专家预取和缓存，显著降低了通常与专家卸载相关的延迟开销，并提高了性价比。大量的集群实验显示，MoE-Infinity 优于现有的许多系统和方法，将延迟降低了 4-20 倍，部署成本降低了 8 倍以上，适用于各种专家混合。MoE-Infinity 的源代码可以在以下网址公开获取。