Despite their popularity in the field of continuous optimisation,
second-order quasi-Newton methods are challenging to apply in machine learning,
as the Hessian matrix is intractably large. This computational burden is
exacerbated by the need to address non-convexity, for instance by modifying the
Hessian's eigenvalues as in Saddle-Free Newton methods. We propose an
optimisation algorithm which addresses both of these concerns - to our
knowledge, the first efficiently-scalable optimisation algorithm to
asymptotically use the exact (eigenvalue-modified) inverse Hessian. Our method
frames the problem as a series which principally square-roots and inverts the
squared Hessian, then uses it to precondition a gradient vector, all without
explicitly computing or eigendecomposing the Hessian. A truncation of this
infinite series provides a new optimisation algorithm which is scalable and
comparable to other first- and second-order optimisation methods in both
runtime and optimisation performance. We demonstrate this in a variety of
settings, including a ResNet-18 trained on CIFAR-10.

提出了一个既能解决大规模的 Hessian 矩阵问题，又能优化非凸性的优化算法，采用了一个无限级数截断的方法，并在多种情境下进行了验证，包括在 CIFAR-10 上训练的 ResNet-18 模型。

神经网络可行的无鞍牛顿优化的 Hessian-Vector 乘积系列

Series of Hessian-Vector Products for Tractable Saddle-Free Newton  Optimisation of Neural Networks

Neural network models are one of the most successful approaches to machine
learning, enjoying an enormous amount of development and research over recent
years and finding concrete real-world applications in almost any conceivable
area of science, engineering and modern life in general. The theoretical
understanding of neural networks trails significantly behind their practical
success and the engineering heuristics that have grown up around them. Random
matrix theory provides a rich framework of tools with which aspects of neural
network phenomenology can be explored theoretically. In this thesis, we
establish significant extensions of prior work using random matrix theory to
understand and describe the loss surfaces of large neural networks,
particularly generalising to different architectures. Informed by the
historical applications of random matrix theory in physics and elsewhere, we
establish the presence of local random matrix universality in real neural
networks and then utilise this as a modeling assumption to derive powerful and
novel results about the Hessians of neural network loss surfaces and their
spectra. In addition to these major contributions, we make use of random matrix
models for neural network loss surfaces to shed light on modern neural network
training approaches and even to derive a novel and effective variant of a
popular optimisation algorithm.
Overall, this thesis provides important contributions to cement the place of
random matrix theory in the theoretical study of modern neural networks,
reveals some of the limits of existing approaches and begins the study of an
entirely new role for random matrix theory in the theory of deep learning with
important experimental discoveries and novel theoretical results based on local
random matrix universality.

此篇论文利用随机矩阵理论来理解和描述神经网络的损失曲面及其谱，并应用于现代神经网络的训练方法，为随机矩阵理论在现代神经网络的理论研究中提供了重要的贡献。

随机矩阵理论与神经网络的损失曲面

Random matrix theory and the loss surfaces of neural networks

Neural networks dominate the modern machine learning landscape, but their
training and success still suffer from sensitivity to empirical choices of
hyperparameters such as model architecture, loss function, and optimisation
algorithm. In this work we present \emph{Population Based Training (PBT)}, a
simple asynchronous optimisation algorithm which effectively utilises a fixed
computational budget to jointly optimise a population of models and their
hyperparameters to maximise performance. Importantly, PBT discovers a schedule
of hyperparameter settings rather than following the generally sub-optimal
strategy of trying to find a single fixed set to use for the whole course of
training. With just a small modification to a typical distributed
hyperparameter training framework, our method allows robust and reliable
training of models. We demonstrate the effectiveness of PBT on deep
reinforcement learning problems, showing faster wall-clock convergence and
higher final performance of agents by optimising over a suite of
hyperparameters. In addition, we show the same method can be applied to
supervised learning for machine translation, where PBT is used to maximise the
BLEU score directly, and also to training of Generative Adversarial Networks to
maximise the Inception score of generated images. In all cases PBT results in
the automatic discovery of hyperparameter schedules and model selection which
results in stable training and better final performance.

本文提出了 Population Based Training 算法，该算法通过优化神经网络模型和超参数的选择，以最大化模型性能，并自动发现超参数的调整进度表以及模型选择。该算法在深度强化学习领域得到了成功的应用。