We introduce the sparse modern Hopfield model as a sparse extension of the
modern Hopfield model. Like its dense counterpart, the sparse modern Hopfield
model equips a memory-retrieval dynamics whose one-step approximation
corresponds to the sparse attention mechanism. Theoretically, our key
contribution is a principled derivation of a closed-form sparse Hopfield energy
using the convex conjugate of the sparse entropic regularizer. Building upon
this, we derive the sparse memory retrieval dynamics from the sparse energy
function and show its one-step approximation is equivalent to the
sparse-structured attention. Importantly, we provide a sparsity-dependent
memory retrieval error bound which is provably tighter than its dense analog.
The conditions for the benefits of sparsity to arise are therefore identified
and discussed. In addition, we show that the sparse modern Hopfield model
maintains the robust theoretical properties of its dense counterpart, including
rapid fixed point convergence and exponential memory capacity. Empirically, we
use both synthetic and real-world datasets to demonstrate that the sparse
Hopfield model outperforms its dense counterpart in many situations.

稀疏的现代 Hopfield 模型通过基于凸共轭的稀疏熵正则化函数推导出闭合形式的稀疏 Hopfield 能量，从而实现了稀疏注意机制和记忆检索动态，并证明了其一步近似等价于稀疏结构的注意力机制，此外，提供了稀疏度相关的记忆检索误差界，表明稀疏性的优势和条件，并表明稀疏的现代 Hopfield 模型保持了其密集对应物的强大理论性能，包括快速固定点收敛和指数级的存储容量。在实验上，使用合成和真实数据集证明稀疏 Hopfield 模型在许多情况下优于其密集对应物。