Weight-sharing is ubiquitous in deep learning. Motivated by this, we
introduce ''weight-sharing regularization'' for neural networks, defined as
$R(w) = \frac{1}{d - 1}\sum_{i > j}^d |w_i - w_j|$. We study the proximal
mapping of $R$ and provide an intuitive interpretation of it in terms of a
physical system of interacting particles. Using this interpretation, we design
a novel parallel algorithm for $\operatorname{prox}_R$ which provides an
exponential speedup over previous algorithms, with a depth of $O(\log^3 d)$.
Our algorithm makes it feasible to train weight-sharing regularized deep neural
networks with proximal gradient descent. Experiments reveal that weight-sharing
regularization enables fully-connected networks to learn convolution-like
filters.

通过引入 'weight-sharing regularization' 为神经网络定义了目标函数，并研究了其近端映射，同时介绍了解释为相互作用粒子的物理系统观点。基于该观点，设计了一种新的并行算法用于近端映射，相较于之前的算法具有指数级加速度，深度为 O (log^3 d)。通过该算法，可以使用近端梯度下降训练采用 weight-sharing regularization 的深度神经网络。实验表明，weight-sharing regularization 能够使全连接网络学习类似卷积操作的滤波器。

权值共享正则化

Weight-Sharing Regularization

In this paper, we propose the Bi-Sub-Gradient (Bi-SG) method, which is a
generalization of the classical sub-gradient method to the setting of convex
bi-level optimization problems. This is a first-order method that is very easy
to implement in the sense that it requires only a computation of the associated
proximal mapping or a sub-gradient of the outer non-smooth objective function,
in addition to a proximal gradient step on the inner optimization problem. We
show, under very mild assumptions, that Bi-SG tackles bi-level optimization
problems and achieves sub-linear rates both in terms of the inner and outer
objective functions. Moreover, if the outer objective function is additionally
strongly convex (still could be non-smooth), the outer rate can be improved to
a linear rate. Last, we prove that the distance of the generated sequence to
the set of optimal solutions of the bi-level problem converges to zero.

本文提出了一种一阶 Bi-Sub-Gradient 方法，它被广泛应用于凸性双层优化问题中，具有易于实现、内外两个目标函数均能取得亚线性收敛速度等特点，并且证明了所生成的序列与双层问题最优解的距离趋于零。

具有非光滑外目标函数的凸双层优化问题

Convex Bi-Level Optimization Problems with Non-smooth Outer Objective  Function

The paper presents a fully explicit algorithm for monotone variational
inequalities. The method uses variable stepsizes that are computed using two
previous iterates as an approximation of the local Lipschitz constant without
running a linesearch. Thus, each iteration of the method requires only one
evaluation of a monotone operator $F$ and a proximal mapping $g$. The operator
$F$ need not be Lipschitz-continuous, which also makes the algorithm
interesting in the area of composite minimization where one cannot use the
descent lemma. The method exhibits an ergodic $O(1/k)$ convergence rate and
$R$-linear rate, if $F, g$ satisfy the error bound condition. We discuss
possible applications of the method to fixed point problems. We discuss
possible applications of the method to fixed point problems as well as its
different generalizations.

本文介绍了一种针对单调变分不等式问题的全显式算法，该算法使用计算变量步长的方法来近似本地 Lipschitz 常数，每次迭代只需要评估单调算子 $F$ 和一个 proximal 映射 $g$，并且如果 $F，g$ 满足误差条件，则该方法表现出的收敛速度是 $O (1/k)$ 和 $R$ 线性速度，可以应用于不同类型的固定点问题。

变分不等式的黄金比例算法

Golden Ratio Algorithms for Variational Inequalities

We consider the problem of minimizing the sum of two convex functions: one is
the average of a large number of smooth component functions, and the other is a
general convex function that admits a simple proximal mapping. We assume the
whole objective function is strongly convex. Such problems often arise in
machine learning, known as regularized empirical risk minimization. We propose
and analyze a new proximal stochastic gradient method, which uses a multi-stage
scheme to progressively reduce the variance of the stochastic gradient. While
each iteration of this algorithm has similar cost as the classical stochastic
gradient method (or incremental gradient method), we show that the expected
objective value converges to the optimum at a geometric rate. The overall
complexity of this method is much lower than both the proximal full gradient
method and the standard proximal stochastic gradient method.

提出了一种多级方案来逐渐减少随机梯度方差的新的近端随机梯度方法，用于解决平滑组件函数的平均和简单近端映射的一般凸函数总和的极小化问题。