This paper presents RAYEN, a framework to impose hard convex constraints on
the output or latent variable of a neural network. RAYEN guarantees that, for
any input or any weights of the network, the constraints are satisfied at all
times. Compared to other approaches, RAYEN does not perform a
computationally-expensive orthogonal projection step onto the feasible set,
does not rely on soft constraints (which do not guarantee the satisfaction of
the constraints at test time), does not use conservative approximations of the
feasible set, and does not perform a potentially slow inner gradient descent
correction to enforce the constraints. RAYEN supports any combination of
linear, convex quadratic, second-order cone (SOC), and linear matrix inequality
(LMI) constraints, achieving a very small computational overhead compared to
unconstrained networks. For example, it is able to impose 1K quadratic
constraints on a 1K-dimensional variable with an overhead of less than 8 ms,
and an LMI constraint with 300x300 dense matrices on a 10K-dimensional variable
in less than 12 ms. When used in neural networks that approximate the solution
of constrained optimization problems, RAYEN achieves computation times between
20 and 7468 times faster than state-of-the-art algorithms, while guaranteeing
the satisfaction of the constraints at all times and obtaining a cost very
close to the optimal one.

本文介绍了 RAYEN 框架，该框架能够在神经网络的输出或潜变量上施加硬凸约束，支持任意线性、凸二次、二阶锥和线性矩阵不等式约束，有效提高了性能并保证在所有时刻满足约束条件

RAYEN: 对神经网络强制采用硬凸约束

RAYEN: Imposition of Hard Convex Constraints on Neural Networks

Sparse Principal Component Analysis is a cardinal technique for obtaining
combinations of features, or principal components (PCs), that explain the
variance of high-dimensional datasets in an interpretable manner. At its heart,
this involves solving a sparsity and orthogonality constrained convex
maximization problem, which is extremely computationally challenging. Most
existing work address sparse PCA via heuristics such as iteratively computing
one sparse PC and deflating the covariance matrix, which does not guarantee the
orthogonality, let alone the optimality, of the resulting solution. We
challenge this status by reformulating the orthogonality conditions as rank
constraints and optimizing over the sparsity and rank constraints
simultaneously. We design tight semidefinite relaxations and propose tractable
second-order cone versions of these relaxations which supply high-quality upper
bounds. We also design valid second-order cone inequalities which hold when
each PC's individual sparsity is specified, and demonstrate that these
inequalities tighten our relaxations significantly. Moreover, we propose exact
methods and rounding mechanisms that exploit these relaxations' tightness to
obtain solutions with a bound gap on the order of 1%-5% for real-world datasets
with p = 100s or 1000s of features and r \in {2, 3} components. We investigate
the performance of our methods in spiked covariance settings and demonstrate
that simultaneously considering the orthogonality and sparsity constraints
leads to improvements in the Area Under the ROC curve of 2%-8% compared to
state-of-the-art deflation methods. All in all, our approach solves sparse PCA
problems with multiple components to certifiable (near) optimality in a
practically tractable fashion.

本研究提出一种新的方法，通过将正交性条件重新表述为秩约束，并同时优化稀疏性和秩约束，使得稀疏主成分分析问题更易解决。通过设计合理的半定松弛和可行的二阶锥不等式，本文的方法在实际数据集中可以获得最优解，并且相比现有方法具有更好的性能。