With the growth of model and data sizes, a broad effort has been made to
design pruning techniques that reduce the resource demand of deep learning
pipelines, while retaining model performance. In order to reduce both inference
and training costs, a prominent line of work uses low-rank matrix
factorizations to represent the network weights. Although able to retain
accuracy, we observe that low-rank methods tend to compromise model robustness
against adversarial perturbations. By modeling robustness in terms of the
condition number of the neural network, we argue that this loss of robustness
is due to the exploding singular values of the low-rank weight matrices. Thus,
we introduce a robust low-rank training algorithm that maintains the network's
weights on the low-rank matrix manifold while simultaneously enforcing
approximate orthonormal constraints. The resulting model reduces both training
and inference costs while ensuring well-conditioning and thus better
adversarial robustness, without compromising model accuracy. This is shown by
extensive numerical evidence and by our main approximation theorem that shows
the computed robust low-rank network well-approximates the ideal full model,
provided a highly performing low-rank sub-network exists.

介绍了一种在保证模型精度的情况下，同时降低深度学习中推理和训练成本，并通过基于神经网络条件数的模型稳健性表明低秩矩阵分解（low-rank matrix factorizations）容易导致模型鲁棒性的问题，提出了一种基于鲁棒低秩矩阵训练的算法，通过施加近似正交约束，保证了低秩表示与近似完整的模型之间精度。通过实验证明降低深度学习模型中推理和训练成本的同时，提高了模型的鲁棒性。

通过近似正交约束的鲁棒低秩训练

Robust low-rank training via approximate orthonormal constraints

What learning algorithms can be run directly on compressively-sensed data? In
this work, we consider the question of accurately and efficiently computing
low-rank matrix or tensor factorizations given data compressed via random
projections. We examine the approach of first performing factorization in the
compressed domain, and then reconstructing the original high-dimensional
factors from the recovered (compressed) factors. In both the matrix and tensor
settings, we establish conditions under which this natural approach will
provably recover the original factors. While it is well-known that random
projections preserve a number of geometric properties of a dataset, our work
can be viewed as showing that they can also preserve certain solutions of
non-convex, NP-Hard problems like non-negative matrix factorization. We support
these theoretical results with experiments on synthetic data and demonstrate
the practical applicability of compressed factorization on real-world gene
expression and EEG time series datasets.

本文考虑在通过随机投影压缩的数据上准确高效地计算低秩矩阵或张量分解的问题。我们研究了在压缩域内执行分解并从恢复 (压缩) 因子中重构原始高维因子的方法，在矩阵和张量设置中，我们建立了这种自然方法能够证明恢复原始因子的条件。我们对合成数据进行了实验，证实了压缩因式分解在真实世界的基因表达和脑电时间序列数据集中的实际适用性，并支持这些理论结果。