Massive Over-activation Yielded Uplifts(MOYU) is an inherent property of
large language models, and dynamic activation(DA) based on the MOYU property is
a clever yet under-explored strategy designed to accelerate inference in these
models. Existing methods that utilize MOYU often face a significant 'Impossible
Trinity': struggling to simultaneously maintain model performance, enhance
inference speed, and extend applicability across various architectures. Due to
the theoretical ambiguities surrounding MOYU, this paper elucidates the root
cause of the MOYU property and outlines the mechanisms behind two primary
limitations encountered by current DA methods: 1) history-related activation
uncertainty, and 2) semantic-irrelevant activation inertia. Our analysis not
only underscores the limitations of current dynamic activation strategies
within large-scale LLaMA models but also proposes opportunities for refining
the design of future sparsity schemes.

大规模语言模型的 Massive Over-activation Yielded Uplifts (MOYU) 是一种固有属性，基于 MOYU 属性的动态激活 (DA) 是一种旨在加速这些模型推理的巧妙但未被充分探索的策略。本文澄清了 MOYU 特性的根本原因，阐述了当前 DA 方法存在的两个主要限制机制：1）与历史相关的激活不确定性，2）语义无关的激活惯性。我们的分析不仅突出了当前大规模 LLaMA 模型中动态激活策略的局限性，还提出了改进未来稀疏度方案设计的机会。