Transformer models, despite their impressive performance, often face
practical limitations due to their high computational requirements. At the same
time, previous studies have revealed significant activation sparsity in these
models, indicating the presence of redundant computations. In this paper, we
propose Dynamic Sparsified Transformer Inference (DSTI), a method that
radically reduces the inference cost of Transformer models by enforcing
activation sparsity and subsequently transforming a dense model into its sparse
Mixture of Experts (MoE) version. We demonstrate that it is possible to train
small gating networks that successfully predict the relative contribution of
each expert during inference. Furthermore, we introduce a mechanism that
dynamically determines the number of executed experts individually for each
token. DSTI can be applied to any Transformer-based architecture and has
negligible impact on the accuracy. For the BERT-base classification model, we
reduce inference cost by almost 60%.

通过激活稀疏性和动态决定每个令牌的执行专家数量的方法，Dynamic Sparsified Transformer Inference（DSTI）能够显著降低 Transformer 模型的推理成本，而且对准确性几乎没有影响。