Vision-language pre-trained models have achieved impressive performance on
various downstream tasks. However, their large model sizes hinder their
utilization on platforms with limited computational resources. We find that
directly using smaller pre-trained models and applying magnitude-based pruning
on CLIP models leads to inflexibility and inferior performance. Recent efforts
for VLP compression either adopt uni-modal compression metrics resulting in
limited performance or involve costly mask-search processes with learnable
masks. In this paper, we first propose the Module-wise Pruning Error (MoPE)
metric, accurately assessing CLIP module importance by performance decline on
cross-modal tasks. Using the MoPE metric, we introduce a unified pruning
framework applicable to both pre-training and task-specific fine-tuning
compression stages. For pre-training, MoPE-CLIP effectively leverages knowledge
from the teacher model, significantly reducing pre-training costs while
maintaining strong zero-shot capabilities. For fine-tuning, consecutive pruning
from width to depth yields highly competitive task-specific models. Extensive
experiments in two stages demonstrate the effectiveness of the MoPE metric, and
MoPE-CLIP outperforms previous state-of-the-art VLP compression methods.

本文提出了一个新的压缩框架，使用 Module-wise Pruning Error（MoPE）度量 CLIP 模块的重要性，并将其应用于预训练和任务特定的微调，通过从宽度到深度连续修剪实现高竞争力的任务特定模型，实验证明 MoPE-CLIP 优于现有的视觉 - 语言预训练模型压缩方法。