Given a set of unlabeled images or (image, text) pairs, contrastive learning aims to pre-train an image encoder that can be used as a feature extractor for many downstream tasks. In this work, we propose EncoderMI, the first membership inference method against image encoders pre-trained by contrastive learning. In particular, given an input and a black-box access to an image encoder, EncoderMI aims to infer whether the input is in the training dataset of the image encoder. EncoderMI can be used 1) by a data owner to audit whether its (public) data was used to pre-train an image encoder without its authorization or 2) by an attacker to compromise privacy of the training data when it is private/sensitive. Our EncoderMI exploits the overfitting of the image encoder towards its training data. In particular, an overfitted image encoder is more likely to output more (or less) similar feature vectors for two augmented versions of an input in (or not in) its training dataset. We evaluate EncoderMI on image encoders pre-trained on multiple datasets by ourselves as well as the Contrastive Language-Image Pre-training (CLIP) image encoder, which is pre-trained on 400 million (image, text) pairs collected from the Internet and released by OpenAI. Our results show that EncoderMI can achieve high accuracy, precision, and recall. We also explore a countermeasure against EncoderMI via preventing overfitting through early stopping. Our results show that it achieves trade-offs between accuracy of EncoderMI and utility of the image encoder, i.e., it can reduce the accuracy of EncoderMI, but it also incurs classification accuracy loss of the downstream classifiers built based on the image encoder.

本文提出了针对对比学习预训练的图像编码器的第一个成员推断方法EncoderMI。EncoderMI利用图像编码器向其训练数据的过度拟合，旨在推断输入是是否在图像编码器的训练数据集中，可用于数据所有者审核公共数据是否被授权用于预训练图像编码器，也可用于攻击者破坏私有/敏感的训练数据的隐私。通过本文，我们还探索了防止过拟合的 EncoderMI 对策，即通过提前停止防止过拟合，虽然会影响 EncoderMI 的准确性，但它将产生基于图像编码器构建的下游分类器的分类准确度损失。

EncoderMI: 对比学习中预训练编码器的成员推断攻击