In this paper, we introduce a privacy-preserving stable diffusion framework leveraging homomorphic encryption, called HE-Diffusion, which primarily focuses on protecting the denoising phase of the diffusion process. HE-Diffusion is a tailored encryption framework specifically designed to align with the unique architecture of stable diffusion, ensuring both privacy and functionality. To address the inherent computational challenges, we propose a novel min-distortion method that enables efficient partial image encryption, significantly reducing the overhead without compromising the model's output quality. Furthermore, we adopt a sparse tensor representation to expedite computational operations, enhancing the overall efficiency of the privacy-preserving diffusion process. We successfully implement HE-based privacy-preserving stable diffusion inference. The experimental results show that HE-Diffusion achieves 500 times speedup compared with the baseline method, and reduces time cost of the homomorphically encrypted inference to the minute level. Both the performance and accuracy of the HE-Diffusion are on par with the plaintext counterpart. Our approach marks a significant step towards integrating advanced cryptographic techniques with state-of-the-art generative models, paving the way for privacy-preserving and efficient image generation in critical applications.

本研究论文介绍了一种称为HE-Diffusion的基于同态加密的隐私保护稳定扩散框架，主要关注扩散过程的去噪阶段的保护。我们提出了一种新颖的最小失真方法，通过有效的部分图像加密显著减少了开销，同时不损害模型的输出质量。此外，我们采用了稀疏张量表示来加快计算操作，提高了隐私保护扩散过程的整体效率。实验结果表明，HE-Diffusion相比基准方法实现了500倍加速，并将同态加密推断的时间成本降低到分钟级别。HE-Diffusion的性能和准确性与明文对应物相媲美。本研究将先进的密码技术与最先进的生成模型相结合，为关键应用领域中的隐私保护和高效图像生成铺平了道路。

使用同态加密的隐私保护扩散模型