While large scale pre-training has achieved great achievements in bridging
the gap between vision and language, it still faces several challenges. First,
the cost for pre-training is expensive. Second, there is no efficient way to
handle the data noise which degrades model performance. Third, previous methods
only leverage limited image-text paired data, while ignoring richer
single-modal data, which may result in poor generalization to single-modal
downstream tasks. In this work, we propose an EfficientCLIP method via Ensemble
Confident Learning to obtain a less noisy data subset. Extra rich non-paired
single-modal text data is used for boosting the generalization of text branch.
We achieve the state-of-the-art performance on Chinese cross-modal retrieval
tasks with only 1/10 training resources compared to CLIP and WenLan, while
showing excellent generalization to single-modal tasks, including text
retrieval and text classification.

本文提出了 EfficientCLIP 方法，通过集成自信度学习来获取数据噪声较少的子集，并利用额外的丰富的非配对单模态文本数据来增强文本分支的泛化能力，从而实现了仅使用 CLIP 和 WenLan 的 1/10 培训资源就能达到中文跨模态检索任务的最优表现，并对文本检索和文本分类等单模态任务表现出良好的泛化能力。