This paper investigates the knowledge of language models from the perspective of Bayesian epistemology. We explore how language models adjust their confidence and responses when presented with evidence with varying levels of informativeness and reliability. To study these properties, we create a dataset with various types of evidence and analyze language models' responses and confidence using verbalized confidence, token probability, and sampling. We observed that language models do not consistently follow Bayesian epistemology: language models follow the Bayesian confirmation assumption well with true evidence but fail to adhere to other Bayesian assumptions when encountering different evidence types. Also, we demonstrated that language models can exhibit high confidence when given strong evidence, but this does not always guarantee high accuracy. Our analysis also reveals that language models are biased toward golden evidence and show varying performance depending on the degree of irrelevance, helping explain why they deviate from Bayesian assumptions.

本文从贝叶斯认识论的角度探讨语言模型的知识，研究了语言模型如何在不同信息量和可靠性的证据面前调整其信心和回应。研究表明，语言模型在面对真实证据时能够较好地遵循贝叶斯确认假设，但在不同类型证据下未能完全遵循其它贝叶斯假设，这种表现差异可能源于模型对黄金证据的偏见及其依赖于相关性程度。

从证据到信念：一种贝叶斯认识论方法研究语言模型