Reconstructing samples from the training set of trained neural networks is a
major privacy concern. Haim et al. (2022) recently showed that it is possible
to reconstruct training samples from neural network binary classifiers, based
on theoretical results about the implicit bias of gradient methods. In this
work, we present several improvements and new insights over this previous work.
As our main improvement, we show that training-data reconstruction is possible
in the multi-class setting and that the reconstruction quality is even higher
than in the case of binary classification. Moreover, we show that using
weight-decay during training increases the vulnerability to sample
reconstruction. Finally, while in the previous work the training set was of
size at most $1000$ from $10$ classes, we show preliminary evidence of the
ability to reconstruct from a model trained on $5000$ samples from $100$
classes.

本文讨论了神经网络训练集的隐私问题。作者对 Haim 等人的研究进行了改进，提出了在多类情况下对训练数据进行重建的方法，并表明使用权重衰减技术训练的模型更容易被重建。此外本研究还展示了从 100 类训练集中的 5000 个样本进行重建的初步证据。

从多类神经网络中重建训练数据

Reconstructing Training Data from Multiclass Neural Networks

We present a theoretically grounded approach to train deep neural networks,
including recurrent networks, subject to class-dependent label noise. We
propose two procedures for loss correction that are agnostic to both
application domain and network architecture. They simply amount to at most a
matrix inversion and multiplication, provided that we know the probability of
each class being corrupted into another. We further show how one can estimate
these probabilities, adapting a recent technique for noise estimation to the
multi-class setting, and thus providing an end-to-end framework. Extensive
experiments on MNIST, IMDB, CIFAR-10, CIFAR-100 and a large scale dataset of
clothing images employing a diversity of architectures --- stacking dense,
convolutional, pooling, dropout, batch normalization, word embedding, LSTM and
residual layers --- demonstrate the noise robustness of our proposals.
Incidentally, we also prove that, when ReLU is the only non-linearity, the loss
curvature is immune to class-dependent label noise.

该论文提出了一种基于理论的方法来训练深度神经网络，包括循环网络，使其适用于存在类别相关标签噪声的情况，并提出两种提高模型噪声稳健性的损失函数矫正方法和一种端到端的噪声估计框架并进行了大量实验证明了这个方法的实用性和有效性。