We propose a new algorithm for efficiently solving the damped Fisher matrix
in large-scale scenarios where the number of parameters significantly exceeds
the number of available samples. This problem is fundamental for natural
gradient descent and stochastic reconfiguration. Our algorithm is based on
Cholesky decomposition and is generally applicable. Benchmark results show that
the algorithm is significantly faster than existing methods.

我们提出了一种新的算法，用于在参数数量明显超过可用样本数量的大规模场景中高效解决阻尼 Fisher 矩阵的问题，这对于自然梯度下降和随机重新构建来说非常重要。我们的算法基于 Cholesky 分解，具有广泛的适用性，并且基准结果表明该算法比现有方法快得多。

大规模阻尼自然梯度下降的高效数值算法

Efficient Numerical Algorithm for Large-Scale Damped Natural Gradient  Descent

Kernel methods provide a principled way to perform non linear, nonparametric
learning. They rely on solid functional analytic foundations and enjoy optimal
statistical properties. However, at least in their basic form, they have
limited applicability in large scale scenarios because of stringent
computational requirements in terms of time and especially memory. In this
paper, we take a substantial step in scaling up kernel methods, proposing
FALKON, a novel algorithm that allows to efficiently process millions of
points. FALKON is derived combining several algorithmic principles, namely
stochastic subsampling, iterative solvers and preconditioning. Our theoretical
analysis shows that optimal statistical accuracy is achieved requiring
essentially $O(n)$ memory and $O(n\sqrt{n})$ time. An extensive experimental
analysis on large scale datasets shows that, even with a single machine, FALKON
outperforms previous state of the art solutions, which exploit
parallel/distributed architectures.

本文介绍了一种名为 FALKON 的算法，该算法可以在大规模场景下有效地处理数百万个数据点，通过结合多种算法原则实现了优化的统计准确性，并采用随机子采样、迭代求解和预处理等算法原则。