We consider the problem of privacy protection in Reinforcement Learning (RL)
algorithms that operate over population processes, a practical but understudied
setting that includes, for example, the control of epidemics in large
populations of dynamically interacting individuals. In this setting, the RL
algorithm interacts with the population over $T$ time steps by receiving
population-level statistics as state and performing actions which can affect
the entire population at each time step. An individual's data can be collected
across multiple interactions and their privacy must be protected at all times.
We clarify the Bayesian semantics of Differential Privacy (DP) in the presence
of correlated data in population processes through a Pufferfish Privacy
analysis. We then give a meta algorithm that can take any RL algorithm as input
and make it differentially private. This is achieved by taking an approach that
uses DP mechanisms to privatize the state and reward signal at each time step
before the RL algorithm receives them as input. Our main theoretical result
shows that the value-function approximation error when applying standard RL
algorithms directly to the privatized states shrinks quickly as the population
size and privacy budget increase. This highlights that reasonable
privacy-utility trade-offs are possible for differentially private RL
algorithms in population processes. Our theoretical findings are validated by
experiments performed on a simulated epidemic control problem over large
population sizes.

我们考虑了强化学习算法中的隐私保护问题，该问题涉及人群过程，这是一个实际但研究很少的设置，例如在大规模个体之间进行流行病控制。我们通过 Pufferfish 隐私分析澄清了相关数据在人群进程中的差分隐私（DP）的贝叶斯语义，并提供了一种元算法，可以使用任何强化学习算法作为输入，并使其具有差分隐私。我们的主要理论结果表明，在隐私化状态下将标准强化学习算法直接应用于状态时，值函数逼近误差会随着人群规模和隐私预算的增加而迅速缩小，从而强调在人群过程中差分隐私强化学习算法可以实现合理的隐私 - 效用平衡。我们的理论发现通过在大规模人口上进行的模拟流行病控制问题的实验证实了。