This article presents a new polynomial parameterized sigmoid called SIGTRON,
which is an extended asymmetric sigmoid with Perceptron, and its companion
convex model called SIGTRON-imbalanced classification (SIC) model that employs
a virtual SIGTRON-induced convex loss function. In contrast to the conventional
$\pi$-weighted cost-sensitive learning model, the SIC model does not have an
external $\pi$-weight on the loss function but has internal parameters in the
virtual SIGTRON-induced loss function. As a consequence, when the given
training dataset is close to the well-balanced condition, we show that the
proposed SIC model is more adaptive to variations of the dataset, such as the
inconsistency of the scale-class-imbalance ratio between the training and test
datasets. This adaptation is achieved by creating a skewed hyperplane equation.
Additionally, we present a quasi-Newton optimization(L-BFGS) framework for the
virtual convex loss by developing an interval-based bisection line search.
Empirically, we have observed that the proposed approach outperforms
$\pi$-weighted convex focal loss and balanced classifier LIBLINEAR(logistic
regression, SVM, and L2SVM) in terms of test classification accuracy with $51$
two-class and $67$ multi-class datasets. In binary classification problems,
where the scale-class-imbalance ratio of the training dataset is not
significant but the inconsistency exists, a group of SIC models with the best
test accuracy for each dataset (TOP$1$) outperforms LIBSVM(C-SVC with RBF
kernel), a well-known kernel-based classifier.

本文介绍了一种名为 SIGTRON 的新的多项式参数化 Sigmoid 函数，并提出了其伴随的凸模型 SIC 模型，该模型使用了一种虚拟的 SIGTRON 诱导的凸损失函数。与传统的加权成本敏感学习模型不同，SIC 模型在损失函数上没有外部的加权，但在虚拟的 SIGTRON 诱导损失函数中有内部参数。因此，当给定的训练数据集接近良好平衡的条件时，我们发现所提出的 SIC 模型对数据集的变化更适应，例如训练集和测试集之间的规模类不平衡比的不一致性。此适应性是通过创建一个倾斜的超平面方程来实现的。此外，我们通过开发一种基于区间的二分线搜索，提出了虚拟凸损失的拟牛顿优化（L-BFGS）框架。从实证上看，我们观察到所提议的方法在 51 个二分类和 67 个多类别数据集的测试分类准确性方面优于加权的凸焦点损失和平衡分类器 LIBLINEAR（逻辑回归、SVM 和 L2SVM）。在二分类问题中，当训练数据集的规模类不平衡比不显著但存在不一致性时，一组针对每个数据集（TOP1）具有最佳测试准确性的 SIC 模型优于 LIBSVM（基于核函数的分类器）。

用于不平衡线性分类的扩展非对称 S 型神经元（SIGTRON）

An extended asymmetric sigmoid with Perceptron (SIGTRON) for imbalanced  linear classification

We consider the development of practical stochastic quasi-Newton, and in
particular Kronecker-factored block-diagonal BFGS and L-BFGS methods, for
training deep neural networks (DNNs). In DNN training, the number of variables
and components of the gradient $n$ is often of the order of tens of millions
and the Hessian has $n^2$ elements. Consequently, computing and storing a full
$n \times n$ BFGS approximation or storing a modest number of (step, change in
gradient) vector pairs for use in an L-BFGS implementation is out of the
question. In our proposed methods, we approximate the Hessian by a
block-diagonal matrix and use the structure of the gradient and Hessian to
further approximate these blocks, each of which corresponds to a layer, as the
Kronecker product of two much smaller matrices. This is analogous to the
approach in KFAC, which computes a Kronecker-factored block-diagonal
approximation to the Fisher matrix in a stochastic natural gradient method.
Because the indefinite and highly variable nature of the Hessian in a DNN, we
also propose a new damping approach to keep the upper as well as the lower
bounds of the BFGS and L-BFGS approximations bounded. In tests on autoencoder
feed-forward neural network models with either nine or thirteen layers applied
to three datasets, our methods outperformed or performed comparably to KFAC and
state-of-the-art first-order stochastic methods.

本文提出了一种使用 Kronecker 乘积近似 Hessian 矩阵和结构化梯度的 Kronecker 分块对角线 BFGS 和 L-BFGS 方法用于深度神经网络训练，通过测试验证其性能优于或与 KFAC 和一阶随机方法相当。

训练深度神经网络的实用拟牛顿方法

Practical Quasi-Newton Methods for Training Deep Neural Networks

The standard L-BFGS method relies on gradient approximations that are not
dominated by noise, so that search directions are descent directions, the line
search is reliable, and quasi-Newton updating yields useful quadratic models of
the objective function. All of this appears to call for a full batch approach,
but since small batch sizes give rise to faster algorithms with better
generalization properties, L-BFGS is currently not considered an algorithm of
choice for large-scale machine learning applications. One need not, however,
choose between the two extremes represented by the full batch or highly
stochastic regimes, and may instead follow a progressive batching approach in
which the sample size increases during the course of the optimization. In this
paper, we present a new version of the L-BFGS algorithm that combines three
basic components - progressive batching, a stochastic line search, and stable
quasi-Newton updating - and that performs well on training logistic regression
and deep neural networks. We provide supporting convergence theory for the
method.

本文介绍了一种结合渐进批处理、随机线搜索和稳定拟牛顿更新的新型 L-BFGS 算法，并提供了该方法在训练逻辑回归和深度神经网络时的收敛理论依据。

机器学习的渐进批处理 L-BFGS 方法

A Progressive Batching L-BFGS Method for Machine Learning

Recurrent Neural Networks (RNNs) are powerful models that achieve exceptional
performance on several pattern recognition problems. However, the training of
RNNs is a computationally difficult task owing to the well-known
"vanishing/exploding" gradient problem. Algorithms proposed for training RNNs
either exploit no (or limited) curvature information and have cheap
per-iteration complexity, or attempt to gain significant curvature information
at the cost of increased per-iteration cost. The former set includes
diagonally-scaled first-order methods such as ADAGRAD and ADAM, while the
latter consists of second-order algorithms like Hessian-Free Newton and K-FAC.
In this paper, we present adaQN, a stochastic quasi-Newton algorithm for
training RNNs. Our approach retains a low per-iteration cost while allowing for
non-diagonal scaling through a stochastic L-BFGS updating scheme. The method
uses a novel L-BFGS scaling initialization scheme and is judicious in storing
and retaining L-BFGS curvature pairs. We present numerical experiments on two
language modeling tasks and show that adaQN is competitive with popular RNN
training algorithms.

本篇论文提出了一种名为 adaQN 的随机拟牛顿算法，用于解决循环神经网络 (RNN) 训练中的梯度消失 / 爆炸问题，该方法使用了一种新的 L-BFGS 缩放初始化方案，并且在存储和保留 L-BFGS 曲率对方面非常明智，实践表明 adaQN 与流行的 RNN 训练算法有相当的竞争力。