We study the optimal memorization capacity of modern Hopfield models and Kernelized Hopfield Models (KHMs), a transformer-compatible class of Dense Associative Memories. We present a tight analysis by establishing a connection between the memory configuration of KHMs and spherical codes from information theory. Specifically, we treat the stored memory set as a specialized spherical code. This enables us to cast the memorization problem in KHMs into a point arrangement problem on a hypersphere. We show that the optimal capacity of KHMs occurs when the feature space allows memories to form an optimal spherical code. This unique perspective leads to: (i) An analysis of how KHMs achieve optimal memory capacity, and identify corresponding necessary conditions. Importantly, we establish an upper capacity bound that matches the well-known exponential lower bound in the literature. This provides the first tight and optimal asymptotic memory capacity for modern Hopfield models. (ii) A sub-linear time algorithm $\mathtt{U}\text{-}\mathtt{Hop}$+ to reach KHMs' optimal capacity. (iii) An analysis of the scaling behavior of the required feature dimension relative to the number of stored memories. These efforts improve both the retrieval capability of KHMs and the representation learning of corresponding transformers. Experimentally, we provide thorough numerical results to back up theoretical findings.

本研究针对现代霍普菲尔德模型及其核化版本（KHMs）在最优记忆容量方面的研究，填补了相关理论空白。通过将KHMs的记忆配置与信息论中的球面码建立联系，提出了一种新颖的方法，将记忆问题转化为超球面上的点排列问题。研究发现KHMs的最优容量在特征空间允许记忆形成最佳球面码时达到，提供了首个紧致且最优的渐近记忆容量分析。

现代霍普菲尔德模型的可证明最佳记忆容量：与变换器兼容的致密关联记忆作为球面码