The few-shot fine-tuning of Latent Diffusion Models (LDMs) has enabled them
to grasp new concepts from a limited number of images. However, given the vast
amount of personal images accessible online, this capability raises critical
concerns about civil privacy. While several previous defense methods have been
developed to prevent such misuse of LDMs, they typically assume that the
textual prompts used by data protectors exactly match those employed by data
exploiters. In this paper, we first empirically demonstrate that breaking this
assumption, i.e., in cases where discrepancies exist between the textual
conditions used by protectors and exploiters, could substantially reduce the
effectiveness of these defenses. Furthermore, considering the visual encoder's
independence from textual prompts, we delve into the visual encoder and
thoroughly investigate how manipulating the visual encoder affects the few-shot
fine-tuning process of LDMs. Drawing on these insights, we propose a simple yet
effective method called \textbf{Prompt-Independent Defense (PID)} to safeguard
privacy against LDMs. We show that PID can act as a strong privacy shield on
its own while requiring significantly less computational power. We believe our
studies, along with the comprehensive understanding and new defense method,
provide a notable advance toward reliable data protection against LDMs.

通过对视觉编码器的操作，提出了名为 Prompt-Independent Defense（PID）的简单而有效的方法，以保护数据免受 Latent Diffusion Models（LDMs）的侵害，并显著减少计算资源的需求。