Training machine learning models with differential privacy (DP) has received increasing interest in recent years. One of the most popular algorithms for training differentially private models is differentially private stochastic gradient descent (DPSGD) and its variants, where at each step gradients are clipped and combined with some noise. Given the increasing usage of DPSGD, we ask the question: is DPSGD alone sufficient to find a good minimizer for every dataset under privacy constraints? As a first step towards answering this question, we show that even for the simple case of linear classification, unlike non-private optimization, (private) feature preprocessing is vital for differentially private optimization. In detail, we first show theoretically that there exists an example where without feature preprocessing, DPSGD incurs a privacy error proportional to the maximum norm of features over all samples. We then propose an algorithm called DPSGD-F, which combines DPSGD with feature preprocessing and prove that for classification tasks, it incurs a privacy error proportional to the diameter of the features $\max_{x, x' \in D} \|x - x'\|_2$. We then demonstrate the practicality of our algorithm on image classification benchmarks.

用差分隐私训练机器学习模型的方法在近年来受到越来越多的关注。我们通过研究线性分类的简单案例，首次展示出在差分隐私优化中，除了使用差分隐私随机梯度下降算法（DPSGD）外，特征预处理对于优化结果的重要性。我们提出了一种名为DPSGD-F的算法，结合了DPSGD和特征预处理，并证明了对于分类任务，它的隐私误差与特征的直径成正比。我们还通过图像分类基准测试展示了我们算法的实用性。

特征预处理对差分隐私线性优化的重要性