Large language models (LLMs) specializing in natural language generation
(NLG) have recently started exhibiting promising capabilities across a variety
of domains. However, gauging the trustworthiness of responses generated by LLMs
remains an open challenge, with limited research on uncertainty quantification
for NLG. Furthermore, existing literature typically assumes white-box access to
language models, which is becoming unrealistic either due to the closed-source
nature of the latest LLMs or due to computational constraints. In this work, we
investigate uncertainty quantification in NLG for $\textit{black-box}$ LLMs. We
first differentiate two closely-related notions: $\textit{uncertainty}$, which
depends only on the input, and $\textit{confidence}$, which additionally
depends on the generated response. We then propose and compare several
confidence/uncertainty metrics, applying them to $\textit{selective NLG}$,
where unreliable results could either be ignored or yielded for further
assessment. Our findings on several popular LLMs and datasets reveal that a
simple yet effective metric for the average semantic dispersion can be a
reliable predictor of the quality of LLM responses. This study can provide
valuable insights for practitioners on uncertainty management when adopting
LLMs. The code to replicate all our experiments is available at
this https URL

该研究探讨了基于黑盒 LLMs 的自然语言生成的不确定性计量，提出几个置信度 / 不确定度统计度量标准，并发现语义分散的平均值可以作为评估 LLMs 响应质量的可靠指标。