Contrastive Language-Image Pre-training (CLIP) model, as an effective
pre-trained multimodal neural network, has been widely used in distributed
machine learning tasks, especially Federated Learning (FL). Typically,
CLIP-based FL adopts Parameter-Efficient Fine-Tuning (PEFT) for model training,
which only fine-tunes adapter parameters or soft prompts rather than the full
parameters. Although PEFT is different from the traditional training mode, in
this paper, we theoretically analyze that the gradients of adapters or soft
prompts can still be used to perform image reconstruction attacks. Based on our
theoretical analysis, we propose Multm-In-Parvo (MIP), a proprietary
reconstruction attack method targeting CLIP-based distributed machine learning
architecture. Specifically, MIP can reconstruct CLIP training images according
to the gradients of soft prompts or an adapter. In addition, MIP includes a
label prediction strategy to accelerate convergence and an inverse gradient
estimation mechanism to avoid the vanishing gradient problem on the text
encoder. Experimental results show that MIP can effectively reconstruct
training images according to the gradients of soft prompts or adapters of CLIP
models.

通过理论分析，本文提出了一种基于 CLIP 模型的多模态分布式机器学习架构下的重构攻击方法 Multm-In-Parvo (MIP)，该方法可以根据软提示或适配器的梯度有效地重构 CLIP 模型的训练图像。