Despite their powerful chat, coding, and reasoning abilities, Large Language
Models (LLMs) frequently hallucinate. Conventional wisdom suggests that
hallucinations are a consequence of a balance between creativity and
factuality, which can be mitigated, but not eliminated, by grounding the LLM in
external knowledge sources. Through extensive systematic experiments, we show
that these traditional approaches fail to explain why LLMs hallucinate in
practice. Specifically, we show that LLMs augmented with a massive Mixture of
Memory Experts (MoME) can easily memorize large datasets of random numbers. We
corroborate these experimental findings with a theoretical construction showing
that simple neural networks trained to predict the next token hallucinate when
the training loss is above a threshold as it usually does in practice when
training on internet scale data. We interpret our findings by comparing against
traditional retrieval methods for mitigating hallucinations. We use our
findings to design a first generation model for removing hallucinations --
Lamini-1 -- that stores facts in a massive mixture of millions of memory
experts that are retrieved dynamically.

通过广泛系统实验，我们展示了传统方法无法解释 LLMs 在实践中为什么会产生幻觉，并通过大量内存专家的混合来增强 LLMs，可以轻松地记忆大数据集，为去除幻觉设计了 Lamini-1 模型。