The storage capacity of a binary classification model is the maximum number
of random input-output pairs per parameter that the model can learn. It is one
of the indicators of the expressive power of machine learning models and is
important for comparing the performance of various models. In this study, we
analyze the structure of the solution space and the storage capacity of fully
connected two-layer neural networks with general activation functions using the
replica method from statistical physics. Our results demonstrate that the
storage capacity per parameter remains finite even with infinite width and that
the weights of the network exhibit negative correlations, leading to a
'division of labor'. In addition, we find that increasing the dataset size
triggers a phase transition at a certain transition point where the permutation
symmetry of weights is broken, resulting in the solution space splitting into
disjoint regions. We identify the dependence of this transition point and the
storage capacity on the choice of activation function. These findings
contribute to understanding the influence of activation functions and the
number of parameters on the structure of the solution space, potentially
offering insights for selecting appropriate architectures based on specific
objectives.

我们通过统计物理学中的复制法，分析具有通用激活函数的全连接双层神经网络的解空间结构和存储容量，并发现存储容量与参数的关系是有限的，在无限宽度的条件下，网络的权重表现出负相关性，并且随着数据集大小的增加，存在一个特定的转变点触发相变，权重的置换对称性被破坏，解空间分裂成不相交的区域。我们进一步确定了这一转变点和存储容量与激活函数的选择之间的依赖关系，这些发现有助于理解激活函数和参数数量对解空间结构的影响，可能为根据特定目标选择合适的架构提供了洞见。

具有通用激活函数的全连接双层神经网络的解空间和存储容量

Solution space and storage capacity of fully connected two-layer neural  networks with generic activation functions

In this paper, we propose to go beyond the well-established approach to
vision-based localization that relies on visual descriptor matching between a
query image and a 3D point cloud. While matching keypoints via visual
descriptors makes localization highly accurate, it has significant storage
demands, raises privacy concerns and requires update to the descriptors in the
long-term. To elegantly address those practical challenges for large-scale
localization, we present GoMatch, an alternative to visual-based matching that
solely relies on geometric information for matching image keypoints to maps,
represented as sets of bearing vectors. Our novel bearing vectors
representation of 3D points, significantly relieves the cross-modal challenge
in geometric-based matching that prevented prior work to tackle localization in
a realistic environment. With additional careful architecture design, GoMatch
improves over prior geometric-based matching work with a reduction of
(10.67m,95.7deg) and (1.43m, 34.7deg) in average median pose errors on
Cambridge Landmarks and 7-Scenes, while requiring as little as 1.5/1.7% of
storage capacity in comparison to the best visual-based matching methods. This
confirms its potential and feasibility for real-world localization and opens
the door to future efforts in advancing city-scale visual localization methods
that do not require storing visual descriptors.

本文提出 GoMatch 作为大规模本地化中仅基于几何信息匹配图像关键点到地图的替代方案，使用 3D 点的旋转矢量表示地图，减少了存储需求并提高了实时性。实验表明，与最佳视觉匹配方法相比，GoMatch 在 Cambridge Landmarks 和 7-Scenes 上的平均姿态中位误差分别降低了 (10.67 米，95.7 度) 和 (1.43 米，34.7 度)，并且只需最佳视觉匹配方法存储能力的 1.5/1.7%。

几何是否足够用于视觉定位中的匹配？

Is Geometry Enough for Matching in Visual Localization?

In [7] Krotov and Hopfield suggest a generalized version of the well-known
Hopfield model of associative memory. In their version they consider a
polynomial interaction function and claim that this increases the storage
capacity of the model. We prove this claim and take the "limit" as the degree
of the polynomial becomes infinite, i.e. an exponential interaction function.
With this interaction we prove that model has an exponential storage capacity
in the number of neurons, yet the basins of attraction are almost as large as
in the standard Hopfield model.

该论文使用多项式交互函数推广了著名的 Hopfield 模型，证明了该模型的存储能力随多项式次数增加呈指数级增长，并且具有与标准 Hopfield 模型几乎相等的吸引域大小。

具有大容量存储的联想记忆模型

On a model of associative memory with huge storage capacity

Data shuffling is one of the fundamental building blocks for distributed
learning algorithms, that increases the statistical gain for each step of the
learning process. In each iteration, different shuffled data points are
assigned by a central node to a distributed set of workers to perform local
computations, which leads to communication bottlenecks. The focus of this paper
is on formalizing and understanding the fundamental information-theoretic
trade-off between storage (per worker) and the worst-case communication
overhead for the data shuffling problem. We completely characterize the
information theoretic trade-off for $K=2$, and $K=3$ workers, for any value of
storage capacity, and show that increasing the storage across workers can
reduce the communication overhead by leveraging coding. We propose a novel and
systematic data delivery and storage update strategy for each data shuffle
iteration, which preserves the structural properties of the storage across the
workers, and aids in minimizing the communication overhead in subsequent data
shuffling iterations.

本文研究数据重排问题中存储和最坏情况通信开销之间的信息熵权衡，并提出了一种新的系统数据传递和存储更新策略以最小化通信开销。