Multiple-choice questions (MCQs) are widely used in the evaluation of large
language models (LLMs) due to their simplicity and efficiency. However, there
are concerns about whether MCQs can truly measure LLM's capabilities,
particularly in knowledge-intensive scenarios where long-form generation (LFG)
answers are required. The misalignment between the task and the evaluation
method demands a thoughtful analysis of MCQ's efficacy, which we undertake in
this paper by evaluating nine LLMs on four question-answering (QA) datasets in
two languages: Chinese and English. We identify a significant issue: LLMs
exhibit an order sensitivity in bilingual MCQs, favoring answers located at
specific positions, i.e., the first position. We further quantify the gap
between MCQs and long-form generation questions (LFGQs) by comparing their
direct outputs, token logits, and embeddings. Our results reveal a relatively
low correlation between answers from MCQs and LFGQs for identical questions.
Additionally, we propose two methods to quantify the consistency and confidence
of LLMs' output, which can be generalized to other QA evaluation benchmarks.
Notably, our analysis challenges the idea that the higher the consistency, the
greater the accuracy. We also find MCQs to be less reliable than LFGQs in terms
of expected calibration error. Finally, the misalignment between MCQs and LFGQs
is not only reflected in the evaluation performance but also in the embedding
space. Our code and models can be accessed at
this https URL

在这篇论文中，我们评估了九个大语言模型在两种语言（中文和英文）的四个问答数据集上的表现，发现大语言模型在双语的多选题中存在一种顺序敏感性，竞争第一位置的选项更容易被选择，与此同时我们提出了两种衡量大语言模型输出一致性和置信度的方法，并发现多选题相较长篇生成题在一致性和预期校准误差方面较不可靠。