Large-scale text-to-image diffusion models excel in generating high-quality
images from textual inputs, yet concerns arise as research indicates their
tendency to memorize and replicate training data, raising We also addressed the
issue of memorization in diffusion models, where models tend to replicate exact
training samples raising copyright infringement and privacy issues. Efforts
within the text-to-image community to address memorization explore causes such
as data duplication, replicated captions, or trigger tokens, proposing
per-prompt inference-time or training-time mitigation strategies. In this
paper, we focus on the feed-forward layers and begin by contrasting neuron
activations of a set of memorized and non-memorized prompts. Experiments reveal
a surprising finding: many different sets of memorized prompts significantly
activate a common subspace in the model, demonstrating, for the first time,
that memorization in the diffusion models lies in a special subspace.
Subsequently, we introduce a novel post-hoc method for editing pre-trained
models, whereby memorization is mitigated through the straightforward pruning
of weights in specialized subspaces, avoiding the need to disrupt the training
or inference process as seen in prior research. Finally, we demonstrate the
robustness of the pruned model against training data extraction attacks,
thereby unveiling new avenues for a practical and one-for-all solution to
memorization.

大规模的文本到图像扩散模型在从文本输入中生成高质量图像方面表现出色，然而研究表明它们倾向于记忆和复制训练数据，引起了版权侵犯和隐私问题。本文针对扩散模型中的记忆问题，通过剪枝特定子空间的权重提出了一种后处理方法，避免了破坏训练或推理过程，从而展示了一个实用且全面的解决记忆问题的新途径。同时展示了经剪枝的模型对于训练数据提取攻击的鲁棒性，揭示了新的解决记忆问题的可能性。