In artificial neural networks, learning from data is a computationally
demanding task in which a large number of connection weights are iteratively
tuned through stochastic-gradient-based heuristic processes over a
cost-function. It is not well understood how learning occurs in these systems,
in particular how they avoid getting trapped in configurations with poor
computational performance. Here we study the difficult case of networks with
discrete weights, where the optimization landscape is very rough even for
simple architectures, and provide theoretical and numerical evidence of the
existence of rare - but extremely dense and accessible - regions of
configurations in the network weight space. We define a novel measure, which we
call the "robust ensemble" (RE), which suppresses trapping by isolated
configurations and amplifies the role of these dense regions. We analytically
compute the RE in some exactly solvable models, and also provide a general
algorithmic scheme which is straightforward to implement: define a
cost-function given by a sum of a finite number of replicas of the original
cost-function, with a constraint centering the replicas around a driving
assignment. To illustrate this, we derive several powerful new algorithms,
ranging from Markov Chains to message passing to gradient descent processes,
where the algorithms target the robust dense states, resulting in substantial
improvements in performance. The weak dependence on the number of precision
bits of the weights leads us to conjecture that very similar reasoning applies
to more conventional neural networks. Analogous algorithmic schemes can also be
applied to other optimization problems.

研究了具有离散权重的人工神经网络中学习过程的优化，提出了一种被称为 “鲁棒集合” 的新量度来抑制被隔离的配置所困住，并分析了稀有但极度密集和可访问的网络配置区域的存在。为解决此问题，提出了一个算法方案来定义目标，并证明了该算法在从驱动分配的稠密状态中提取信息时的有效性。