Training machine learning models with differential privacy (DP) has received
increasing interest in recent years. One of the most popular algorithms for
training differentially private models is differentially private stochastic
gradient descent (DPSGD) and its variants, where at each step gradients are
clipped and combined with some noise. Given the increasing usage of DPSGD, we
ask the question: is DPSGD alone sufficient to find a good minimizer for every
dataset under privacy constraints? As a first step towards answering this
question, we show that even for the simple case of linear classification,
unlike non-private optimization, (private) feature preprocessing is vital for
differentially private optimization. In detail, we first show theoretically
that there exists an example where without feature preprocessing, DPSGD incurs
a privacy error proportional to the maximum norm of features over all samples.
We then propose an algorithm called DPSGD-F, which combines DPSGD with feature
preprocessing and prove that for classification tasks, it incurs a privacy
error proportional to the diameter of the features $\max_{x, x' \in D} \|x -
x'\|_2$. We then demonstrate the practicality of our algorithm on image
classification benchmarks.

用差分隐私训练机器学习模型的方法在近年来受到越来越多的关注。我们通过研究线性分类的简单案例，首次展示出在差分隐私优化中，除了使用差分隐私随机梯度下降算法（DPSGD）外，特征预处理对于优化结果的重要性。我们提出了一种名为 DPSGD-F 的算法，结合了 DPSGD 和特征预处理，并证明了对于分类任务，它的隐私误差与特征的直径成正比。我们还通过图像分类基准测试展示了我们算法的实用性。