Adaptive optimization methods have become the default solvers for many machine learning tasks. Unfortunately, the benefits of adaptivity may degrade when training with differential privacy, as the noise added to ensure privacy reduces the effectiveness of the adaptive preconditioner. To this end, we propose AdaDPS, a general framework that uses non-sensitive side information to precondition the gradients, allowing the effective use of adaptive methods in private settings. We formally show AdaDPS reduces the amount of noise needed to achieve similar privacy guarantees, thereby improving optimization performance. Empirically, we leverage simple and readily available side information to explore the performance of AdaDPS in practice, comparing to strong baselines in both centralized and federated settings. Our results show that AdaDPS improves accuracy by 7.7% (absolute) on average -- yielding state-of-the-art privacy-utility trade-offs on large-scale text and image benchmarks.

AdaDPS是一种通用的框架，使用非敏感侧面信息对梯度进行预处理，可以在私有环境中有效地使用自适应方法，降低噪音所需的数量，从而改善优化性能。在大规模文本和图像基准测试中，相对于强基线，AdaDPS平均提高了7.7％（绝对）的准确度，实现了最先进的隐私效用权衡。

带辅助信息的私有自适应优化