Neural network models are one of the most successful approaches to machine
learning, enjoying an enormous amount of development and research over recent
years and finding concrete real-world applications in almost any conceivable
area of science, engineering and modern life in general. The theoretical
understanding of neural networks trails significantly behind their practical
success and the engineering heuristics that have grown up around them. Random
matrix theory provides a rich framework of tools with which aspects of neural
network phenomenology can be explored theoretically. In this thesis, we
establish significant extensions of prior work using random matrix theory to
understand and describe the loss surfaces of large neural networks,
particularly generalising to different architectures. Informed by the
historical applications of random matrix theory in physics and elsewhere, we
establish the presence of local random matrix universality in real neural
networks and then utilise this as a modeling assumption to derive powerful and
novel results about the Hessians of neural network loss surfaces and their
spectra. In addition to these major contributions, we make use of random matrix
models for neural network loss surfaces to shed light on modern neural network
training approaches and even to derive a novel and effective variant of a
popular optimisation algorithm.
Overall, this thesis provides important contributions to cement the place of
random matrix theory in the theoretical study of modern neural networks,
reveals some of the limits of existing approaches and begins the study of an
entirely new role for random matrix theory in the theory of deep learning with
important experimental discoveries and novel theoretical results based on local
random matrix universality.

此篇论文利用随机矩阵理论来理解和描述神经网络的损失曲面及其谱，并应用于现代神经网络的训练方法，为随机矩阵理论在现代神经网络的理论研究中提供了重要的贡献。

随机矩阵理论与神经网络的损失曲面

Random matrix theory and the loss surfaces of neural networks

Federated learning is a distributed machine learning approach which enables a
shared server model to learn by aggregating the locally-computed parameter
updates with the training data from spatially-distributed client silos. Though
successfully possessing advantages in both scale and privacy, federated
learning is hurt by domain shift problems, where the learning models are unable
to generalize to unseen domains whose data distribution is non-i.i.d. with
respect to the training domains. In this study, we propose the Federated
Invariant Learning Consistency (FedILC) approach, which leverages the gradient
covariance and the geometric mean of Hessians to capture both inter-silo and
intra-silo consistencies of environments and unravel the domain shift problems
in federated networks. The benchmark and real-world dataset experiments bring
evidence that our proposed algorithm outperforms conventional baselines and
similar federated learning algorithms. This is relevant to various fields such
as medical healthcare, computer vision, and the Internet of Things (IoT). The
code is released at this https URL.

本研究提出了 FedILC 方法，利用梯度协方差和 Hessians 的几何平均来捕捉环境内部和环境之间的一致性，解决联邦网络中的域漂移问题，并在基准和真实世界数据集的实验证明了我们的算法优于传统基准和类似的联邦学习算法。

FedILC: 非独立同分布数据联邦学习中加权几何均值和不变梯度协方差

FedILC: Weighted Geometric Mean and Invariant Gradient Covariance for Federated Learning on Non-IID Data

We show that Newton's method converges globally at a linear rate for
objective functions whose Hessians are stable. This class of problems includes
many functions which are not strongly convex, such as logistic regression. Our
linear convergence result is (i) affine-invariant, and holds even if an (ii)
approximate Hessian is used, and if the subproblems are (iii) only solved
approximately. Thus we theoretically demonstrate the superiority of Newton's
method over first-order methods, which would only achieve a sublinear
$O(1/t^2)$ rate under similar conditions.

证明了牛顿法对于具有稳定 Hessians 的目标函数具有全局收敛的线性速度，在这一类问题中包括了许多不是强凸的函数，如逻辑回归，相比于仅在类似条件下实现次线性 $O（1/t^2）$ 收敛率的一阶方法，我们的线性收敛结果是（i）仿射不变的，即使使用（ii）近似海森和（iii）仅近似解决子问题。