Deep neural network models, though very powerful and highly successful, are
computationally expensive in terms of space and time. Recently, there have been
a number of attempts on binarizing the network weights and activations. This
greatly reduces the network size, and replaces the underlying multiplications
to additions or even XNOR bit operations. However, existing binarization
schemes are based on simple matrix approximation and ignore the effect of
binarization on the loss. In this paper, we propose a proximal Newton algorithm
with diagonal Hessian approximation that directly minimizes the loss w.r.t. the
binarized weights. The underlying proximal step has an efficient closed-form
solution, and the second-order information can be efficiently obtained from the
second moments already computed by the Adam optimizer. Experiments on both
feedforward and recurrent networks show that the proposed loss-aware
binarization algorithm outperforms existing binarization schemes, and is also
more robust for wide and deep networks.

本文介绍了一种基于近端牛顿算法的深度神经网络权重二值化方法，能够直接最小化权重二值化时的损失函数，相对于现有的方法表现更好，尤其适用于深度网络和循环神经网络。

深度网络的损失感知量化

Loss-aware Binarization of Deep Networks

We propose an algorithmic framework for convex minimization problems of a
composite function with two terms: a self-concordant function and a possibly
nonsmooth regularization term. Our method is a new proximal Newton algorithm
that features a local quadratic convergence rate. As a specific instance of our
framework, we consider the sparse inverse covariance matrix estimation in graph
learning problems. Via a careful dual formulation and a novel analytic
step-size selection procedure, our approach for graph learning avoids Cholesky
decompositions and matrix inversions in its iteration making it attractive for
parallel and distributed implementations.

提出了一种基于新的近端牛顿算法的凸优化方法，应用于图学习问题中的稀疏逆协方差矩阵估计，避免了矩阵求逆，适合并行实现。