Modern Hopfield networks have enjoyed recent interest due to their connection
to attention in transformers. Our paper provides a unified framework for sparse
Hopfield networks by establishing a link with Fenchel-Young losses. The result
is a new family of Hopfield-Fenchel-Young energies whose update rules are
end-to-end differentiable sparse transformations. We reveal a connection
between loss margins, sparsity, and exact memory retrieval. We further extend
this framework to structured Hopfield networks via the SparseMAP
transformation, which can retrieve pattern associations instead of a single
pattern. Experiments on multiple instance learning and text rationalization
demonstrate the usefulness of our approach.

利用 Fenchel-Young 损失函数，我们提出了一种稀疏 Hopfield 网络的统一框架，该网络与 transformers 中的 attention 有关，通过端到端可微分的稀疏变换实现更新规则，进而揭示了损失边界、稀疏性和准确内存检索之间的联系。同时，我们通过 SparseMAP 变换将该框架进一步扩展到结构化 Hopfield 网络，能够检索模式关联而不是单个模式。多实例学习和文本理据化的实验证明了我们方法的有效性。