A key challenge, supported both theoretically and empirically, is that
robustness demands greater network capacity and more data than standard
training. However, effectively adding capacity under stringent Lipschitz
constraints has proven more difficult than it may seem, evident by the fact
that state-of-the-art approach tend more towards \emph{underfitting} than
overfitting. Moreover, we posit that a lack of careful exploration of the
design space for Lipshitz-based approaches has left potential performance gains
on the table. In this work, we provide a more comprehensive evaluation to
better uncover the potential of Lipschitz-based certification methods. Using a
combination of novel techniques, design optimizations, and synthesis of prior
work, we are able to significantly improve the state-of-the-art \emph{verified
robust accuracy} (VRA) for deterministic certification on a variety of
benchmark datasets, and over a range of perturbation sizes. Of particular note,
we discover that the addition of large "Cholesky-orthogonalized residual dense"
layers to the end of existing state-of-the-art Lipschitz-controlled ResNet
architectures is especially effective for increasing network capacity and
performance. Combined with filtered generative data augmentation, our final
results further the state of the art deterministic VRA by up to 8.5 percentage
points. Code is available at https://github.com/hukkai/liresnet.

通过使用新技术、设计优化和先前工作的综合综合，我们能够显著提高确定性认证方法在各种基准数据集上的最新鲜准确性，并扩展了扰动范围大小，尤其是通过在现有最新雅可比约束控制的 ResNet 架构末端添加大型 “Cholesky - 正交化残差稠密” 层，以增加网络容量和性能，再结合过滤的生成数据增强，最终结果比确定性认证方法的最新准确性提高 8.5 个百分点。

提高可认证鲁棒性的配方：容量与数据

A Recipe for Improved Certifiable Robustness: Capacity and Data

Recent studies show that training deep neural networks (DNNs) with Lipschitz
constraints are able to enhance adversarial robustness and other model
properties such as stability. In this paper, we propose a layer-wise orthogonal
training method (LOT) to effectively train 1-Lipschitz convolution layers via
parametrizing an orthogonal matrix with an unconstrained matrix. We then
efficiently compute the inverse square root of a convolution kernel by
transforming the input domain to the Fourier frequency domain. On the other
hand, as existing works show that semi-supervised training helps improve
empirical robustness, we aim to bridge the gap and prove that semi-supervised
learning also improves the certified robustness of Lipschitz-bounded models. We
conduct comprehensive evaluations for LOT under different settings. We show
that LOT significantly outperforms baselines regarding deterministic l2
certified robustness, and scales to deeper neural networks. Under the
supervised scenario, we improve the state-of-the-art certified robustness for
all architectures (e.g. from 59.04% to 63.50% on CIFAR-10 and from 32.57% to
34.59% on CIFAR-100 at radius rho = 36/255 for 40-layer networks). With
semi-supervised learning over unlabelled data, we are able to improve
state-of-the-art certified robustness on CIFAR-10 at rho = 108/255 from 36.04%
to 42.39%. In addition, LOT consistently outperforms baselines on different
model architectures with only 1/3 evaluation time.

该研究提出一种基于层次正交训练和利普希茨限制的深度神经网络训练方法，结合半监督学习来提高模型的认证鲁棒性，并证明其在多个数据集和网络结构上表现出卓越的性能。

LOT: 层级正交训练在提高 L2 认证鲁棒性上的应用

LOT: Layer-wise Orthogonal Training on Improving $\ell_2$ Certified  Robustness

Lipschitz constraints under L2 norm on deep neural networks are useful for
provable adversarial robustness bounds, stable training, and Wasserstein
distance estimation. While heuristic approaches such as the gradient penalty
have seen much practical success, it is challenging to achieve similar
practical performance while provably enforcing a Lipschitz constraint. In
principle, one can design Lipschitz constrained architectures using the
composition property of Lipschitz functions, but Anil et al. recently
identified a key obstacle to this approach: gradient norm attenuation. They
showed how to circumvent this problem in the case of fully connected networks
by designing each layer to be gradient norm preserving. We extend their
approach to train scalable, expressive, provably Lipschitz convolutional
networks. In particular, we present the Block Convolution Orthogonal
Parameterization (BCOP), an expressive parameterization of orthogonal
convolution operations. We show that even though the space of orthogonal
convolutions is disconnected, the largest connected component of BCOP with 2n
channels can represent arbitrary BCOP convolutions over n channels. Our BCOP
parameterization allows us to train large convolutional networks with provable
Lipschitz bounds. Empirically, we find that it is competitive with existing
approaches to provable adversarial robustness and Wasserstein distance
estimation.

通过 Block Convolution 正交参数化的方法，我们可以训练具有可证明的 Lipschitz 界限的大型卷积神经网络，其表现与现有方法相当，实用性强。

Lipschitz 约束卷积神经网络中梯度消失的预防

Preventing Gradient Attenuation in Lipschitz Constrained Convolutional  Networks

In this paper, we study the convergence of generative adversarial networks
(GANs) from the perspective of the informativeness of the gradient of the
optimal discriminative function. We show that GANs without restriction on the
discriminative function space commonly suffer from the problem that the
gradient produced by the discriminator is uninformative to guide the generator.
By contrast, Wasserstein GAN (WGAN), where the discriminative function is
restricted to 1-Lipschitz, does not suffer from such a gradient
uninformativeness problem. We further show in the paper that the model with a
compact dual form of Wasserstein distance, where the Lipschitz condition is
relaxed, may also theoretically suffer from this issue. This implies the
importance of Lipschitz condition and motivates us to study the general
formulation of GANs with Lipschitz constraint, which leads to a new family of
GANs that we call Lipschitz GANs (LGANs). We show that LGANs guarantee the
existence and uniqueness of the optimal discriminative function as well as the
existence of a unique Nash equilibrium. We prove that LGANs are generally
capable of eliminating the gradient uninformativeness problem. According to our
empirical analysis, LGANs are more stable and generate consistently higher
quality samples compared with WGAN.

本文从判别函数最优梯度含义化角度研究生成对抗网络 (GANs) 的收敛性，并表明通过添加 Lipschitz 约束可以消除由于梯度缺乏信息而导致的问题，因此提出了一类名为 Lipschitz GANs (LGANs) 的 GANs，实验证明 LGANs 的样本质量相较于 Wasserstein GANs 更高且更加稳定。