Generative models, such as DALL-E, Midjourney, and Stable Diffusion, have
societal implications that extend beyond the field of computer science. These
models require large image databases like LAION-2B, which contain two billion
images. At this scale, manual inspection is difficult and automated analysis is
challenging. In addition, recent studies show that duplicated images pose
copyright problems for models trained on LAION2B, which hinders its usability.
This paper proposes an algorithmic chain that runs with modest compute, that
compresses CLIP features to enable efficient duplicate detection, even for vast
image volumes. Our approach demonstrates that roughly 700 million images, or
about 30\%, of LAION-2B's images are likely duplicated. Our method also
provides the histograms of duplication on this dataset, which we use to reveal
more examples of verbatim copies by Stable Diffusion and further justify the
approach. The current version of the de-duplicated set will be distributed
online.

本文提出了一种算法链，能够对规模为 20 亿张图片的 LAION-2B 数据集进行有效的重复检测，并揭示了大量的版权问题，能够充分解决当前模型训练时所遇到的问题。

关于 LAION-2B 数据去重的研究

On the De-duplication of LAION-2B

Image denoising can be described as the problem of mapping from a noisy image
to a noise-free image. The best currently available denoising methods
approximate this mapping with cleverly engineered algorithms. In this work we
attempt to learn this mapping directly with plain multi layer perceptrons (MLP)
applied to image patches. We will show that by training on large image
databases we are able to outperform the current state-of-the-art image
denoising methods. In addition, our method achieves results that are superior
to one type of theoretical bound and goes a large way toward closing the gap
with a second type of theoretical bound. Our approach is easily adapted to less
extensively studied types of noise, such as mixed Poisson-Gaussian noise, JPEG
artifacts, salt-and-pepper noise and noise resembling stripes, for which we
achieve excellent results as well. We will show that combining a block-matching
procedure with MLPs can further improve the results on certain images. In a
second paper, we detail the training trade-offs and the inner mechanisms of our
MLPs.

本文中，我们使用简单的多层感知器（MLP）应用于图像块来直接学习从嘈杂的图像到无噪声图像的映射，并通过训练大型图像数据库，我们能够超越当前最先进的图像降噪方法并获得优秀的成果，同时在其他类型的噪声方面也取得了出色的结果，此外，结合块匹配程序可以进一步提高性能。