We introduce BSDetector, a method for detecting bad and speculative answers
from a pretrained Large Language Model by estimating a numeric confidence score
for any output it generated. Our uncertainty quantification technique works for
any LLM accessible only via a black-box API, and combines intrinsic and
extrinsic assessments of confidence into a single trustworthiness estimate for
any LLM response to a given prompt. Our method is extremely general and can
applied to all of the best LLMs available today (whose training data remains
unknown). By expending a bit of extra computation, users of any LLM API can now
get the same response as they would ordinarily, as well as a confidence
estimate that caution when not to trust this response. Experiments on both
closed and open-form Question-Answer benchmarks reveal that BSDetector more
accurately identifies incorrect LLM responses than alternative uncertainty
estimation procedures (for both GPT-3 and ChatGPT). By sampling multiple
responses from the LLM and considering the one with the highest confidence
score, we can additionally obtain more accurate responses from the same LLM,
without any extra training steps.

BSDetector 通过估计预训练的大型语言模型生成的任何输出的数值置信度得分，引入了检测不良和推测性答案的方法。通过采样多个 LLM 的响应并考虑最高置信度得分的响应，我们还可以从相同的 LLM 获取更准确的响应，而无需任何额外的训练步骤。