The widespread adoption of Large Language Models (LLMs) has become
commonplace, particularly with the emergence of open-source models. More
importantly, smaller models are well-suited for integration into consumer
devices and are frequently employed either as standalone solutions or as
subroutines in various AI tasks. Despite their ubiquitous use, there is no
systematic analysis of their specific capabilities and limitations. In this
study, we tackle one of the most widely used tasks - answering Multiple Choice
Question (MCQ). We analyze 26 small open-source models and find that 65% of the
models do not understand the task, only 4 models properly select an answer from
the given choices, and only 5 of these models are choice order independent.
These results are rather alarming given the extensive use of MCQ tests with
these models. We recommend exercising caution and testing task understanding
before using MCQ to evaluate LLMs in any field whatsoever.

通过分析 26 个小型开源模型，发现 65% 的模型不能理解任务，只有 4 个模型能正确选择答案，其中只有 5 个模型对选项顺序无关，这些结果对于广泛使用 MCQ 测试的模型来说相当令人担忧。建议在任何情况下使用 MCQ 评估 LLMs 之前要小心谨慎地测试任务理解能力。