Learning arguably involves the discovery and memorization of abstract rules.
The aim of this paper is to study associative memory mechanisms. Our model is
based on high-dimensional matrices consisting of outer products of embeddings,
which relates to the inner layers of transformer language models. We derive
precise scaling laws with respect to sample size and parameter size, and
discuss the statistical efficiency of different estimators, including
optimization-based algorithms. We provide extensive numerical experiments to
validate and interpret theoretical results, including fine-grained
visualizations of the stored memory associations.

通过学习高维矩阵和内层变换器语言模型，研究关联记忆机制、标度定律和统计效率的优化算法，通过数值实验验证理论结果和存储记忆联想的细粒度可视化。

关于联想记忆的缩放定律

Scaling Laws for Associative Memories

The current paper presents a novel machinery for studying non-asymptotic
minimax estimation of high-dimensional matrices, which yields tight minimax
rates for a large collection of loss functions in a variety of problems.
Based on the convex geometry of finite-dimensional Banach spaces, we first
develop a volume ratio approach for determining minimax estimation rates of
unconstrained normal mean matrices under all squared unitarily invariant norm
losses. In addition, we establish the minimax rates for estimating mean
matrices with submatrix sparsity, where the sparsity constraint introduces an
additional term in the rate whose dependence on the norm differs completely
from the rate of the unconstrained problem. Moreover, the approach is
applicable to the matrix completion problem under the low-rank constraint.
The new method also extends beyond the normal mean model. In particular, it
yields tight rates in covariance matrix estimation and Poisson rate matrix
estimation problems for all unitarily invariant norms.

本文提出了一种用于研究非渐近极小值估计高维矩阵的新机制，该机制可以在各种问题的大量损失函数中产生紧密的极小值。基于有限维 Banach 空间的凸几何，我们首先开发了体积比方法，用于确定所有平方酉不变范数下无约束正常均值矩阵的极小值估计率。此外，我们还建立了具有子矩阵稀疏度的均值矩阵估计的极小值率，其中稀疏性约束引入了一个附加术语，该术语对范数的依赖性与无约束问题的速率完全不同。此外，该方法也适用于低秩约束下的矩阵完成问题。新方法还扩展到正常均值模型之外。特别地，它对于所有酉不变范数的协方差矩阵估计和泊松速率矩阵估计问题都产生了紧密的速率。