Membership inference attacks (MIAs) on diffusion models have emerged as potential evidence of unauthorized data usage in training pre-trained diffusion models. These attacks aim to detect the presence of specific images in training datasets of diffusion models. Our study delves into the evaluation of state-of-the-art MIAs on diffusion models and reveals critical flaws and overly optimistic performance estimates in existing MIA evaluation. We introduce CopyMark, a more realistic MIA benchmark that distinguishes itself through the support for pre-trained diffusion models, unbiased datasets, and fair evaluation pipelines. Through extensive experiments, we demonstrate that the effectiveness of current MIA methods significantly degrades under these more practical conditions. Based on our results, we alert that MIA, in its current state, is not a reliable approach for identifying unauthorized data usage in pre-trained diffusion models. To the best of our knowledge, we are the first to discover the performance overestimation of MIAs on diffusion models and present a unified benchmark for more realistic evaluation. Our code is available on GitHub: \url{https://github.com/caradryanl/CopyMark}.

本研究针对扩散模型的会员推断攻击（MIA）进行评估，揭示了现有 MIA 评估中的关键缺陷和过于乐观的性能估计。我们提出了 CopyMark，一种更为可靠的 MIA 基准，在更实际的条件下显示出当前 MIA 方法效果显著降低，表明目前的 MIA 不能可靠地识别在扩散模型中未经授权的数据使用。

真实世界基准使会员推断攻击在扩散模型上失败