Artificial neural networks (ANNs) require tremendous amount of data to train
on. However, in classification models, most data features are often similar
which can lead to increase in training time without significant improvement in
the performance. Thus, we hypothesize that there could be a more efficient way
to train an ANN using a better representative sample. For this, we propose the
LAD Improved Iterative Training (LIIT), a novel training approach for ANN using
large deviations principle to generate and iteratively update training samples
in a fast and efficient setting. This is exploratory work with extensive
opportunities for future work. The thesis presents this ongoing research work
with the following contributions from this study: (1) We propose a novel ANN
training method, LIIT, based on the large deviations theory where additional
dimensionality reduction is not needed to study high dimensional data. (2) The
LIIT approach uses a Modified Training Sample (MTS) that is generated and
iteratively updated using a LAD anomaly score based sampling strategy. (3) The
MTS sample is designed to be well representative of the training data by
including most anomalous of the observations in each class. This ensures
distinct patterns and features are learnt with smaller samples. (4) We study
the classification performance of the LIIT trained ANNs with traditional batch
trained counterparts.

本文介绍了一个基于大偏差原理的新型人工神经网络训练方法 LAD Improved Iterative Training，该方法使用 Modified Training Sample（MTS）作为训练样本，采用异常得分为基础的采样策略，能更加有效地学习高维数据特征和模式，与传统批处理方式相比分类性能更优。

加速神经网络训练的大偏差

Large Deviations for Accelerating Neural Networks Training

We interpret the variational inference of the Stochastic Gradient Descent
(SGD) as minimizing a new potential function named the
\textit{quasi-potential}. We analytically construct the quasi-potential
function in the case when the loss function is convex and admits only one
global minimum point. We show in this case that the quasi-potential function is
related to the noise covariance structure of SGD via a partial differential
equation of Hamilton-Jacobi type. This relation helps us to show that
anisotropic noise leads to faster escape than isotropic noise. We then consider
the dynamics of SGD in the case when the loss function is non-convex and admits
several different local minima. In this case, we demonstrate an example that
shows how the noise covariance structure plays a role in "implicit
regularization", a phenomenon in which SGD favors some particular local minimum
points. This is done through the relation between the noise covariance
structure and the quasi-potential function. Our analysis is based on Large
Deviations Theory (LDT), and they are validated by numerical experiments.

本研究利用变分推理和大偏差理论对随机梯度下降进行分析，提出了一个新的潜在函数模型，并探讨了噪声协方差结构对 SGD 的影响及其在隐式正则化中的作用。