Prompting and Multiple Choices Questions (MCQ) have become the preferred
approach to assess the capabilities of Large Language Models (LLMs), due to
their ease of manipulation and evaluation. Such experimental appraisals have
pointed toward the LLMs' apparent ability to perform causal reasoning or to
grasp uncertainty. In this paper, we investigate whether these abilities are
measurable outside of tailored prompting and MCQ by reformulating these issues
as direct text completion - the foundation of LLMs. To achieve this goal, we
define scenarios with multiple possible outcomes and we compare the prediction
made by the LLM through prompting (their Stated Answer) to the probability
distributions they compute over these outcomes during next token prediction
(their Revealed Belief). Our findings suggest that the Revealed Belief of LLMs
significantly differs from their Stated Answer and hint at multiple biases and
misrepresentations that their beliefs may yield in many scenarios and outcomes.
As text completion is at the core of LLMs, these results suggest that common
evaluation methods may only provide a partial picture and that more research is
needed to assess the extent and nature of their capabilities.

我们通过转化问题、多选题和直接文本补全的实验评估，研究了大型语言模型（LLMs）在因果推理、不确定性方面能力的量化，结果显示 LLMs 的表态答案与预测真实信念存在显著差异，提示它们的信念可能在多种情景和结果中存在多重偏见和不准确性，对于 LLMs 能力的评估方法仅提供了部分信息，需要进一步研究其能力的广度和本质。