In this paper, we show that the process of continually learning new tasks and memorizing previous tasks introduces unknown privacy risks and challenges to bound the privacy loss. Based upon this, we introduce a formal definition of Lifelong DP, in which the participation of any data tuples in the training set of any tasks is protected, under a consistently bounded DP protection, given a growing stream of tasks. A consistently bounded DP means having only one fixed value of the DP privacy budget, regardless of the number of tasks. To preserve Lifelong DP, we propose a scalable and heterogeneous algorithm, called L2DP-ML with a streaming batch training, to efficiently train and continue releasing new versions of an L2M model, given the heterogeneity in terms of data sizes and the training order of tasks, without affecting DP protection of the private training set. An end-to-end theoretical analysis and thorough evaluations show that our mechanism is significantly better than baseline approaches in preserving Lifelong DP. The implementation of L2DP-ML is available at: https://github.com/haiphanNJIT/PrivateDeepLearning.

本文介绍了一种被称为Lifelong DP的连续隐私定义，提出了一种可扩展和异构的算法，称为L2DP-ML，用于保护在一个不断增长的任务流中任何数据元组的参与，以保护训练集的隐私，并且无论任务数量如何，DP隐私预算都有一个固定的值。实现表明，我们的机制显着优于基线方法，能更好地保护Lifelong DP。

终身DP：在终身机器学习中保持一致的有界差分隐私