Diffusion models (DMs) are widely used for generating high-quality image
datasets. However, since they operate directly in the high-dimensional pixel
space, optimization of DMs is computationally expensive, requiring long
training times. This contributes to large amounts of noise being injected into
the differentially private learning process, due to the composability property
of differential privacy. To address this challenge, we propose training Latent
Diffusion Models (LDMs) with differential privacy. LDMs use powerful
pre-trained autoencoders to reduce the high-dimensional pixel space to a much
lower-dimensional latent space, making training DMs more efficient and fast.
Unlike [Ghalebikesabi et al., 2023] that pre-trains DMs with public data then
fine-tunes them with private data, we fine-tune only the attention modules of
LDMs at varying layers with privacy-sensitive data, reducing the number of
trainable parameters by approximately 96% compared to fine-tuning the entire
DM. We test our algorithm on several public-private data pairs, such as
ImageNet as public data and CIFAR10 and CelebA as private data, and SVHN as
public data and MNIST as private data. Our approach provides a promising
direction for training more powerful, yet training-efficient differentially
private DMs that can produce high-quality synthetic images.

研究训练以潜在空间作为输入的 LDM 的方法，与传统的 Diffusion Models 相比，通过使用预训练的自动编码器将高维像素空间降为低维潜在空间，大大提高了训练效率，并使用隐私保护的方式对不同维度的注意力模型进行训练以减少可调参数，最终的实验结果表明该方法在生成高质量合成图像时表现突出。