The recent work CLIPA presents an inverse scaling law for CLIP training --
whereby the larger the image/text encoders used, the shorter the sequence
length of image/text tokens that can be applied in training. This finding
enables us to train high-performance CLIP models with significantly reduced
computations. Building upon this work, we hereby present CLIPA-v2 with two key
contributions. Technically, we find this inverse scaling law is also applicable
in the finetuning stage, enabling further reduction in computational needs.
Empirically, we explore CLIPA at scale, extending the experiments up to the
H/14 model with ~13B image-text pairs seen during training.
Our results are exciting -- by only allocating a budget of \$10,000, our CLIP
model achieves an impressive zero-shot ImageNet accuracy of 81.1%, surpassing
the prior best CLIP model (from OpenCLIP, 80.1%) by 1.0% and meanwhile reducing
the computational cost by ~39X. Moreover, with an additional investment of
$4,000, we can further elevate the zero-shot ImageNet accuracy to 81.8%. Our
code and models are available at this https URL

通过研究表明，CLIPA 可以通过反比例尺度律以更低的计算成本实现高性能的训练，可以在 finetuning 阶段进一步减少计算需求，从而实现零样本 Imagenet 准确率的提高。

CLIPA-v2: 用 1 万美金预算训练，实现 81.1% 的零样本 ImageNet 准确率；再花 4000 美金可获得 81.8% 的准确率

CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy  within a \$10,000 Budget; An Extra \$4,000 Unlocks 81.8% Accuracy

Large sparsely-activated models have obtained excellent performance in
multiple domains. However, such models are typically trained on a single
modality at a time. We present the Language-Image MoE, LIMoE, a sparse mixture
of experts model capable of multimodal learning. LIMoE accepts both images and
text simultaneously, while being trained using a contrastive loss. MoEs are a
natural fit for a multimodal backbone, since expert layers can learn an
appropriate partitioning of modalities. However, new challenges arise; in
particular, training stability and balanced expert utilization, for which we
propose an entropy-based regularization scheme. Across multiple scales, we
demonstrate remarkable performance improvement over dense models of equivalent
computational cost. LIMoE-L/16 trained comparably to CLIP-L/14 achieves 78.6%
zero-shot ImageNet accuracy (vs. 76.2%), and when further scaled to H/14 (with
additional data) it achieves 84.1%, comparable to state-of-the-art methods
which use larger custom per-modality backbones and pre-training schemes. We
analyse the quantitative and qualitative behavior of LIMoE, and demonstrate
phenomena such as differing treatment of the modalities and the organic
emergence of modality-specific experts.

提出了一种用于多模态学习的稀疏专家模型 LIMoE，该模型接受图像和文本，使用对比损失进行训练，并提出一种熵基准的正则化方案。LIMoE 在多个尺度上展示出优异的性能，并能够实现零样本 ImageNet 准确度的显著提高。