Auto-regressive generation models achieve competitive performance across many
different NLP tasks such as summarization, question answering, and
classifications. However, they are also known for being slow in inference,
which makes them challenging to deploy in real-time applications. We propose a
switchable decision to accelerate inference by dynamically assigning
computation resources for each data instance. Automatically making decisions on
where to skip and how to balance quality and computation cost with constrained
optimization, our dynamic neural generation networks enforce the efficient
inference path and determine the optimized trade-off. Experiments across
question answering, summarization, and classification benchmarks show that our
method benefits from less computation cost during inference while keeping the
same accuracy. Extensive experiments and ablation studies demonstrate that our
method can be general, effective, and beneficial for many NLP tasks.

通过动态分配计算资源来加速推理，我们的动态神经生成网络在问题回答、摘要和分类基准测试中表现出少量的计算成本，同时保持相同的准确性。

可切换的决策：动态神经生成网络

Switchable Decision: Dynamic Neural Generation Networks

We propose a general and efficient framework to control auto-regressive
generation models with NeurAlly-Decomposed Oracle (NADO). Given a pre-trained
base language model and a sequence-level boolean oracle function, we propose to
decompose the oracle function into token-level guidance to steer the base model
in text generation. Specifically, the token-level guidance is approximated by a
neural model trained with examples sampled from the base model, demanding no
additional auxiliary labeled data. Based on posterior regularization, we
present the closed-form optimal solution to incorporate the token-level
guidance into the base model for controllable generation. We further provide a
theoretical analysis of how the approximation quality of NADO affects the
controllable generation results. Experiments conducted on two applications: (1)
text generation with lexical constraints and (2) machine translation with
formality control demonstrate that our framework efficiently guides the base
model towards the given oracle while maintaining high generation quality.

本研究提出基于 NeurAlly-Decomposed Oracle (NADO) 的通用高效生成模型控制框架，其中 NADO 将控制模型分解为词级指导以引导底层模型在文本生成过程中生成符合预期的结果，通过实验证明，本框架在维护高生成质量的同时能高效地指导模型生成出符合特定条件要求的结果。