Safe deployment of large language models (LLMs) may benefit from a reliable
method for assessing their generated content to determine when to abstain or to
selectively generate. While likelihood-based metrics such as perplexity are
widely employed, recent research has demonstrated the limitations of using
sequence-level probability estimates given by LLMs as reliable indicators of
generation quality. Conversely, LLMs have demonstrated strong calibration at
the token level, particularly when it comes to choosing correct answers in
multiple-choice questions or evaluating true/false statements. In this work, we
reformulate open-ended generation tasks into token-level prediction tasks, and
leverage LLMs' superior calibration at the token level. We instruct an LLM to
self-evaluate its answers, employing either a multi-way comparison or a
point-wise evaluation approach, with the option to include a ``None of the
above'' option to express the model's uncertainty explicitly. We benchmark a
range of scoring methods based on self-evaluation and evaluate their
performance in selective generation using TruthfulQA and TL;DR. Through
experiments with PaLM-2 and GPT-3, we demonstrate that self-evaluation based
scores not only improve accuracy, but also correlate better with the overall
quality of generated content.

使用大型语言模型进行自我评估可以提高生成内容的准确性，并与生成内容的整体质量更好地相关。

大型语言模型中自我评估提高选择性生成

Self-Evaluation Improves Selective Generation in Large Language Models

Nowadays, most methods in end-to-end contextual speech recognition bias the
recognition process towards contextual knowledge. Since all-neural contextual
biasing methods rely on phrase-level contextual modeling and attention-based
relevance modeling, they may encounter confusion between similar
context-specific phrases, which hurts predictions at the token level. In this
work, we focus on mitigating confusion problems with fine-grained contextual
knowledge selection (FineCoS). In FineCoS, we introduce fine-grained knowledge
to reduce the uncertainty of token predictions. Specifically, we first apply
phrase selection to narrow the range of phrase candidates, and then conduct
token attention on the tokens in the selected phrase candidates. Moreover, we
re-normalize the attention weights of most relevant phrases in inference to
obtain more focused phrase-level contextual representations, and inject
position information to better discriminate phrases or tokens. On LibriSpeech
and an in-house 160,000-hour dataset, we explore the proposed methods based on
a controllable all-neural biasing method, collaborative decoding (ColDec). The
proposed methods provide at most 6.1% relative word error rate reduction on
LibriSpeech and 16.4% relative character error rate reduction on the in-house
dataset over ColDec.

本篇论文主要探讨利用细粒度上下文知识选择方式，以降低基于 “专有” 短语的上下文建模产生的不确定性问题，并引入位置信息来更好地区分短语或令牌，达到在语音识别上提高识别准确率的效果。

利用细粒度上下文知识选择提升端到端上下文语音识别

Improving End-to-End Contextual Speech Recognition with Fine-Grained Contextual Knowledge Selection

With the success of language pretraining, it is highly desirable to develop
more efficient architectures of good scalability that can exploit the abundant
unlabeled data at a lower cost. To improve the efficiency, we examine the
much-overlooked redundancy in maintaining a full-length token-level
presentation, especially for tasks that only require a single-vector
presentation of the sequence. With this intuition, we propose
Funnel-Transformer which gradually compresses the sequence of hidden states to
a shorter one and hence reduces the computation cost. More importantly, by
re-investing the saved FLOPs from length reduction in constructing a deeper or
wider model, we further improve the model capacity. In addition, to perform
token-level predictions as required by common pretraining objectives,
Funnel-Transformer is able to recover a deep representation for each token from
the reduced hidden sequence via a decoder. Empirically, with comparable or
fewer FLOPs, Funnel-Transformer outperforms the standard Transformer on a wide
variety of sequence-level prediction tasks, including text classification,
language understanding, and reading comprehension. The code and pretrained
checkpoints are available at this https URL

本研究提出了 Funnel-Transformer 模型，利用序列压缩来提升模型效率，在保证单向量序列表征的前提下极大地减少了计算成本。通过在模型构建时重新投资在长度缩短中节省下来的 FLOPs，进一步提高了模型容量。实验表明，Funnel-Transformer 在一系列序列级别的预测任务上表现优秀，包括文本分类、语言理解和阅读理解。