Uncertainty estimation plays a pivotal role in ensuring the reliability of
safety-critical human-AI interaction systems, particularly in the medical
domain. However, a general method for quantifying the uncertainty of free-form
answers has yet to be established in open-ended medical question-answering (QA)
tasks, where irrelevant words and sequences with limited semantic information
can be the primary source of uncertainty due to the presence of generative
inequality. In this paper, we propose the Word-Sequence Entropy (WSE), which
calibrates the uncertainty proportion at both the word and sequence levels
according to the semantic relevance, with greater emphasis placed on keywords
and more relevant sequences when performing uncertainty quantification. We
compare WSE with 6 baseline methods on 5 free-form medical QA datasets,
utilizing 7 "off-the-shelf" large language models (LLMs), and show that WSE
exhibits superior performance on accurate uncertainty measurement under two
standard criteria for correctness evaluation (e.g., WSE outperforms existing
state-of-the-art method by 3.23% AUROC on the MedQA dataset). Additionally, in
terms of the potential for real-world medical QA applications, we achieve a
significant enhancement in the performance of LLMs when employing sequences
with lower uncertainty, identified by WSE, as final answers (e.g., +6.36%
accuracy improvement on the COVID-QA dataset), without requiring any additional
task-specific fine-tuning or architectural modifications.

我们提出了一种 Word-Sequence Entropy (WSE) 的方法，用于在医学问答任务中对不确定性进行量化，通过语义相关性在词和序列级别上校准不确定性比例，在不需额外的任务特定微调或架构修改的情况下，通过 WSE 识别低不确定性的序列作为最终答案，实现在现实世界医学问答应用中的显著性能提升。