Auto-regressive generation models achieve competitive performance across many
different NLP tasks such as summarization, question answering, and
classifications. However, they are also known for being slow in inference,
which makes them challenging to deploy in real-time applications. We propose a
switchable decision to accelerate inference by dynamically assigning
computation resources for each data instance. Automatically making decisions on
where to skip and how to balance quality and computation cost with constrained
optimization, our dynamic neural generation networks enforce the efficient
inference path and determine the optimized trade-off. Experiments across
question answering, summarization, and classification benchmarks show that our
method benefits from less computation cost during inference while keeping the
same accuracy. Extensive experiments and ablation studies demonstrate that our
method can be general, effective, and beneficial for many NLP tasks.

通过动态分配计算资源来加速推理，我们的动态神经生成网络在问题回答、摘要和分类基准测试中表现出少量的计算成本，同时保持相同的准确性。