Large Language Models (LLMs) have demonstrated remarkable capabilities across
various applications, fundamentally reshaping the landscape of natural language
processing (NLP) research. However, recent evaluation frameworks often rely on
the output probabilities of LLMs for predictions, primarily due to
computational constraints, diverging from real-world LLM usage scenarios. While
widely employed, the efficacy of these probability-based evaluation strategies
remains an open research question. This study aims to scrutinize the validity
of such probability-based evaluation methods within the context of using LLMs
for Multiple Choice Questions (MCQs), highlighting their inherent limitations.
Our empirical investigation reveals that the prevalent probability-based
evaluation method inadequately aligns with generation-based prediction.
Furthermore, current evaluation frameworks typically assess LLMs through
predictive tasks based on output probabilities rather than directly generating
responses, owing to computational limitations. We illustrate that these
probability-based approaches do not effectively correspond with generative
predictions. The outcomes of our study can enhance the understanding of LLM
evaluation methodologies and provide insights for future research in this
domain.

使用大型语言模型（LLMs）进行多项选择题（MCQs）的实证研究表明，概率评估方法在生成预测方面存在内在局限性，与当前评估框架通常基于输出概率而非直接生成回应的计算限制相关，结果强调了 LLMs 评估方法的有效性和未来研究的启示。