Recent advances in language models (LMs) have led to significant improvements
in quality on complex NLP tasks, but at the expense of increased inference
costs. Cascading offers a simple strategy to achieve more favorable
cost-quality tradeoffs: here, a small model is invoked for most "easy"
instances, while a few "hard" instances are deferred to the large model. While
the principles underpinning cascading are well-studied for classification tasks
- with deferral based on predicted class uncertainty favored theoretically and
practically - a similar understanding is lacking for generative LM tasks. In
this work, we initiate a systematic study of deferral rules for LM cascades. We
begin by examining the natural extension of predicted class uncertainty to
generative LM tasks, namely, the predicted sequence uncertainty. We show that
this measure suffers from the length bias problem, either over- or
under-emphasizing outputs based on their lengths. This is because LMs produce a
sequence of uncertainty values, one for each output token; and moreover, the
number of output tokens is variable across examples. To mitigate this issue, we
propose to exploit the richer token-level uncertainty information implicit in
generative LMs. We argue that naive predicted sequence uncertainty corresponds
to a simple aggregation of these uncertainties. By contrast, we show that
incorporating token-level uncertainty through learned post-hoc deferral rules
can significantly outperform such simple aggregation strategies, via
experiments on a range of natural language benchmarks with FLAN-T5 models. We
further show that incorporating embeddings from the smaller model and
intermediate layers of the larger model can give an additional boost in the
overall cost-quality tradeoff.

通过对 FLAN-T5 模型在一系列自然语言基准测试中进行实验，我们表明，通过学习后续的推迟规则来结合较小模型的嵌入和较大模型的中间层，可以在整体成本与质量的权衡中实现额外的提升。