We present PSiLON Net, an MLP architecture that uses $L_1$ weight
normalization for each weight vector and shares the length parameter across the
layer. The 1-path-norm provides a bound for the Lipschitz constant of a neural
network and reflects on its generalizability, and we show how PSiLON Net's
design drastically simplifies the 1-path-norm, while providing an inductive
bias towards efficient learning and near-sparse parameters. We propose a
pruning method to achieve exact sparsity in the final stages of training, if
desired. To exploit the inductive bias of residual networks, we present a
simplified residual block, leveraging concatenated ReLU activations. For
networks constructed with such blocks, we prove that considering only a subset
of possible paths in the 1-path-norm is sufficient to bound the Lipschitz
constant. Using the 1-path-norm and this improved bound as regularizers, we
conduct experiments in the small data regime using overparameterized PSiLON
Nets and PSiLON ResNets, demonstrating reliable optimization and strong
performance.

提出了一种名为 PSiLON Net 的 MLP 架构，利用 $L_1$ 权重归一化来处理每个权重向量，并在层间共享长度参数。通过简化 1-path-norm 并提供对高效学习和近稀疏参数的归纳偏差，设计了 PSiLON Net。同时，提出了一种修剪方法来在最后阶段实现精确稀疏性。使用改进的 1-path-norm 作为正则化器，针对小数据环境下过参数化的 PSiLON Nets 和 PSiLON ResNets 进行实验，展示了可靠的优化和强大的性能。

隐藏的协同作用：$L_1$ 权重归一化与 1 - 路径 - 范数正则化

Hidden Synergy: $L_1$ Weight Normalization and 1-Path-Norm  Regularization

We demonstrate two new important properties of the 1-path-norm of shallow
neural networks. First, despite its non-smoothness and non-convexity it allows
a closed form proximal operator which can be efficiently computed, allowing the
use of stochastic proximal-gradient-type methods for regularized empirical risk
minimization. Second, when the activation functions is differentiable, it
provides an upper bound on the Lipschitz constant of the network. Such bound is
tighter than the trivial layer-wise product of Lipschitz constants, motivating
its use for training networks robust to adversarial perturbations. In practical
experiments we illustrate the advantages of using the proximal mapping and we
compare the robustness-accuracy trade-off induced by the 1-path-norm, L1-norm
and layer-wise constraints on the Lipschitz constant (Parseval networks).

研究了浅层神经网络的 1-path - 范数，发现它可以通过闭合形式的近端算子进行计算，且可以提供网络李普希茨常数的上界，是训练鲁棒性强的网络的有力工具。与 L1 - 范数以及基于层的李普希茨常数的制约相比，1-path - 范数的近端映射具有更好的鲁棒性 - 准确性平衡。