We propose an ensemble learning framework with Poisson sub-sampling to
effectively train a collection of teacher models to issue some differential
privacy (DP) guarantee for training data. Through boosting under DP, a student
model derived from the training data suffers little model degradation from the
models trained with no privacy protection. Our proposed solution leverages upon
two mechanisms, namely: (i) a privacy budget amplification via Poisson
sub-sampling to train a target prediction model that requires less noise to
achieve a same level of privacy budget, and (ii) a combination of the
sub-sampling technique and an ensemble teacher-student learning framework that
introduces DP-preserving noise at the output of the teacher models and
transfers DP-preserving properties via noisy labels. Privacy-preserving student
models are then trained with the noisy labels to learn the knowledge with
DP-protection from the teacher model ensemble. Experimental evidences on spoken
command recognition and continuous speech recognition of Mandarin speech show
that our proposed framework greatly outperforms existing DP-preserving
algorithms in both speech processing tasks.

提出了一种使用泊松子采样的集成学习框架，有效地训练一组教师模型以针对训练数据发出一些不同隐私保护（DP）保证。通过在 DP 下 Boosting，从训练数据中派生的学生模型遭受的模型退化很少，可以从无隐私保护的模型中进行训练。