Unsupervised pre-training is a common step in developing computer vision
models and large language models. In this setting, the absence of labels
requires the use of similarity-based loss functions, such as contrastive loss,
that favor minimizing the distance between similar inputs and maximizing the
distance between distinct inputs. As privacy concerns mount, training these
models using differential privacy has become more important. However, due to
how inputs are generated for these losses, one of their undesirable properties
is that their $L_2$ sensitivity can grow with increasing batch size. This
property is particularly disadvantageous for differentially private training
methods, such as DP-SGD. To overcome this issue, we develop a new DP-SGD
variant for similarity based loss functions -- in particular the commonly used
contrastive loss -- that manipulates gradients of the objective function in a
novel way to obtain a senstivity of the summed gradient that is $O(1)$ for
batch size $n$. We test our DP-SGD variant on some preliminary CIFAR-10
pre-training and CIFAR-100 finetuning tasks and show that, in both tasks, our
method's performance comes close to that of a non-private model and generally
outperforms DP-SGD applied directly to the contrastive loss.

使用新的 DP-SGD 变体解决相似性损失函数的梯度敏感度随着批量大小增长而增加的问题，并在 CIFAR-10 预训练和 CIFAR-100 微调任务中得到了与非私有模型接近甚至优于直接应用于相似性损失的 DP-SGD 的性能。