Recent advancements in the capabilities of large language models (LLMs) have
paved the way for a myriad of groundbreaking applications in various fields.
However, a significant challenge arises as these models often "hallucinate",
i.e., fabricate facts without providing users an apparent means to discern the
veracity of their statements. Uncertainty estimation (UE) methods are one path
to safer, more responsible, and more effective use of LLMs. However, to date,
research on UE methods for LLMs has been focused primarily on theoretical
rather than engineering contributions. In this work, we tackle this issue by
introducing LM-Polygraph, a framework with implementations of a battery of
state-of-the-art UE methods for LLMs in text generation tasks, with unified
program interfaces in Python. Additionally, it introduces an extendable
benchmark for consistent evaluation of UE techniques by researchers, and a demo
web application that enriches the standard chat dialog with confidence scores,
empowering end-users to discern unreliable responses. LM-Polygraph is
compatible with the most recent LLMs, including BLOOMz, LLaMA-2, ChatGPT, and
GPT-4, and is designed to support future releases of similarly-styled LMs.

大型语言模型的不确定性估计方法 LM-Polygraph 可为文本生成任务提供可靠的响应，并为研究人员提供一致评估的基准，以帮助终端用户辨别可靠性。

LM-Polygraph：语言模型的不确定性估计

LM-Polygraph: Uncertainty Estimation for Language Models

Several uncertainty estimation methods have been recently proposed for
machine translation evaluation. While these methods can provide a useful
indication of when not to trust model predictions, we show in this paper that
the majority of them tend to underestimate model uncertainty, and as a result
they often produce misleading confidence intervals that do not cover the ground
truth. We propose as an alternative the use of conformal prediction, a
distribution-free method to obtain confidence intervals with a theoretically
established guarantee on coverage. First, we demonstrate that split conformal
prediction can ``correct'' the confidence intervals of previous methods to
yield a desired coverage level. Then, we highlight biases in estimated
confidence intervals, both in terms of the translation language pairs and the
quality of translations. We apply conditional conformal prediction techniques
to obtain calibration subsets for each data subgroup, leading to equalized
coverage.

本文研究机器翻译的不确定性评估方法，发现目前大多数方法对模型不确定性的估计较低，建议采用分布自由的合规预测法来保证覆盖度，并提出条件合规预测技术来获得每个数据子组的校准子集，从而实现覆盖率均衡。

机器翻译评估的共形化

Conformalizing Machine Translation Evaluation

Advances in deep neural network (DNN) based molecular property prediction
have recently led to the development of models of remarkable accuracy and
generalization ability, with graph convolution neural networks (GCNNs)
reporting state-of-the-art performance for this task. However, some challenges
remain and one of the most important that needs to be fully addressed concerns
uncertainty quantification. DNN performance is affected by the volume and the
quality of the training samples. Therefore, establishing when and to what
extent a prediction can be considered reliable is just as important as
outputting accurate predictions, especially when out-of-domain molecules are
targeted. Recently, several methods to account for uncertainty in DNNs have
been proposed, most of which are based on approximate Bayesian inference. Among
these, only a few scale to the large datasets required in applications.
Evaluating and comparing these methods has recently attracted great interest,
but results are generally fragmented and absent for molecular property
prediction. In this paper, we aim to quantitatively compare scalable techniques
for uncertainty estimation in GCNNs. We introduce a set of quantitative
criteria to capture different uncertainty aspects, and then use these criteria
to compare MC-Dropout, deep ensembles, and bootstrapping, both theoretically in
a unified framework that separates aleatoric/epistemic uncertainty and
experimentally on the QM9 dataset. Our experiments quantify the performance of
the different uncertainty estimation methods and their impact on
uncertainty-related error reduction. Our findings indicate that ensembling and
bootstrapping consistently outperform MC-Dropout, with different
context-specific pros and cons. Our analysis also leads to a better
understanding of the role of aleatoric/epistemic uncertainty and highlights the
challenge posed by out-of-domain uncertainty.

本研究旨在定量比较 GCNNs 中不确定性估计的可扩展技术，介绍了一组定量标准来捕捉不同方面的不确定性，并使用这些标准在 QM9 数据集上理论和实验比较了 MC-Dropout、deep ensembles 和 bootstrapping 这些不确定估计方法，结果表明合奏和自举法始终优于 MC-Dropout 方法，在不同的具体情境中具有不同的优点和缺点，并突出了来自不同领域的不确定性所带来的挑战。