Contrastive pretraining of image-text foundation models, such as CLIP,
demonstrated excellent zero-shot performance and improved robustness on a wide
range of downstream tasks. However, these models utilize large
transformer-based encoders with significant memory and latency overhead which
pose challenges for deployment on mobile devices. In this work, we introduce
MobileCLIP -- a new family of efficient image-text models optimized for runtime
performance along with a novel and efficient training approach, namely
multi-modal reinforced training. The proposed training approach leverages
knowledge transfer from an image captioning model and an ensemble of strong
CLIP encoders to improve the accuracy of efficient models. Our approach avoids
train-time compute overhead by storing the additional knowledge in a reinforced
dataset. MobileCLIP sets a new state-of-the-art latency-accuracy tradeoff for
zero-shot classification and retrieval tasks on several datasets. Our
MobileCLIP-S2 variant is 2.3$\times$ faster while more accurate compared to
previous best CLIP model based on ViT-B/16. We further demonstrate the
effectiveness of our multi-modal reinforced training by training a CLIP model
based on ViT-B/16 image backbone and achieving +2.9% average performance
improvement on 38 evaluation benchmarks compared to the previous best.
Moreover, we show that the proposed approach achieves 10$\times$-1000$\times$
improved learning efficiency when compared with non-reinforced CLIP training.

在这项研究中，我们引入了 MobileCLIP 这个新的高效图像 - 文本模型系列，通过一种名为多模态增强训练的新颖且高效的训练方法，利用图像字幕模型和一组强大的 CLIP 编码器中的知识转移，将额外的知识存储在增强数据集中而避免了训练时计算开销，从而实现了零样本分类和检索任务的新的最佳延迟 - 准确性权衡。