State-of-the-art large language models are sometimes distributed as
open-source software but are also increasingly provided as a closed-source
service. These closed-source large-language models typically see the widest
usage by the public, however, they often do not provide an estimate of their
uncertainty when responding to queries. As even the best models are prone to
``hallucinating" false information with high confidence, a lack of a reliable
estimate of uncertainty limits the applicability of these models in critical
settings. We explore estimating the uncertainty of closed-source LLMs via
multiple rephrasings of an original base query. Specifically, we ask the model,
multiple rephrased questions, and use the similarity of the answers as an
estimate of uncertainty. We diverge from previous work in i) providing rules
for rephrasing that are simple to memorize and use in practice ii) proposing a
theoretical framework for why multiple rephrased queries obtain calibrated
uncertainty estimates. Our method demonstrates significant improvements in the
calibration of uncertainty estimates compared to the baseline and provides
intuition as to how query strategies should be designed for optimal test
calibration.

通过多次重新表述原始基本查询以估计封闭源语言模型的不确定性，来提高模型的不确定性校准，并为优化测试校准设计查询策略提供直观指导。

通过多个重新表述的查询来估计封闭源语言模型中的不确定性

Just rephrase it! Uncertainty estimation in closed-source language  models via multiple rephrased queries

ChatGPT has changed the AI community and an active research line is the
performance evaluation of ChatGPT. A key challenge for the evaluation is that
ChatGPT is still closed-source and traditional benchmark datasets may have been
used by ChatGPT as the training data. In this paper, (i) we survey recent
studies which uncover the real performance levels of ChatGPT in seven
categories of NLP tasks, (ii) review the social implications and safety issues
of ChatGPT, and (iii) emphasize key challenges and opportunities for its
evaluation. We hope our survey can shed some light on its blackbox manner, so
that researchers are not misleaded by its surface generation.

我们在这篇论文中调查了 ChatGPT 在七个自然语言处理任务领域的真实性能水平，回顾了 ChatGPT 的社会影响和安全问题，并强调了其评估中的关键挑战和机遇。我们希望我们的调查能够揭示 ChatGPT 的黑盒特性，以免研究人员被其表面生成所误导。