State of the art computer vision models have been shown to be vulnerable to
small adversarial perturbations of the input. In other words, most images in
the data distribution are both correctly classified by the model and are very
close to a visually similar misclassified image. Despite substantial research
interest, the cause of the phenomenon is still poorly understood and remains
unsolved. We hypothesize that this counter intuitive behavior is a naturally
occurring result of the high dimensional geometry of the data manifold. As a
first step towards exploring this hypothesis, we study a simple synthetic
dataset of classifying between two concentric high dimensional spheres. For
this dataset we show a fundamental tradeoff between the amount of test error
and the average distance to nearest error. In particular, we prove that any
model which misclassifies a small constant fraction of a sphere will be
vulnerable to adversarial perturbations of size $O(1/\sqrt{d})$. Surprisingly,
when we train several different architectures on this dataset, all of their
error sets naturally approach this theoretical bound. As a result of the
theory, the vulnerability of neural networks to small adversarial perturbations
is a logical consequence of the amount of test error observed. We hope that our
theoretical analysis of this very simple case will point the way forward to
explore how the geometry of complex real-world data sets leads to adversarial
examples.

研究表明，计算机视觉模型容易受到输入微小对抗扰动攻击，而该现象是由数据流形的高维几何性质引起的，结果表明神经网络的易受小型对抗性扰动攻击是测试误差的逻辑结果。

对抗球体

Adversarial Spheres

Computing the first few singular vectors of a large matrix is a problem that
frequently comes up in statistics and numerical analysis. Given the presence of
noise, exact calculation is hard to achieve, and the following problem is of
importance:
\vskip2mm \centerline {\it How much a small perturbation to the matrix
changes the singular vectors ?} \vskip2mm Answering this question, classical
theorems, such as those of Davis-Kahan and Wedin, give tight estimates for the
worst-case scenario. In this paper, we show that if the perturbation (noise) is
random and our matrix has low rank, then better estimates can be obtained. Our
method relies on high dimensional geometry and is different from those used an
earlier papers.

本文提出了一种新方法，利用高维几何和随机噪声来更好地估计低秩矩阵的奇异向量，以应对矩阵扰动问题。