The success of contrastive language-image pretraining (CLIP) relies on the
supervision from the pairing between images and captions, which tends to be
noisy in web-crawled data. We present Mixture of Data Experts (MoDE) and learn
a system of CLIP data experts via clustering. Each data expert is trained on
one data cluster, being less sensitive to false negative noises in other
clusters. At inference time, we ensemble their outputs by applying weights
determined through the correlation between task metadata and cluster
conditions. To estimate the correlation precisely, the samples in one cluster
should be semantically similar, but the number of data experts should still be
reasonable for training and inference. As such, we consider the ontology in
human language and propose to use fine-grained cluster centers to represent
each data expert at a coarse-grained level. Experimental studies show that four
CLIP data experts on ViT-B/16 outperform the ViT-L/14 by OpenAI CLIP and
OpenCLIP on zero-shot image classification but with less ($<$35\%) training
cost. Meanwhile, MoDE can train all data expert asynchronously and can flexibly
include new data experts. The code is available at
this https URL

使用混合数据专家（MoDE）方法提高神经网络 CLIP 在零样本图像分类任务上的性能，通过聚类学习多个数据专家，使用元数据与聚类条件的相关性确定权重进行模型集成。

MoDE: 通过聚类实现 CLIP 数据专家

MoDE: CLIP Data Experts via Clustering

Web-crawled data provides a good source of parallel corpora for training
machine translation models. It is automatically obtained, but extremely noisy,
and recent work shows that neural machine translation systems are more
sensitive to noise than traditional statistical machine translation methods. In
this paper, we propose a novel approach to filter out noisy sentence pairs from
web-crawled corpora via pre-trained language models. We measure sentence
parallelism by leveraging the multilingual capability of BERT and use the
Generative Pre-training (GPT) language model as a domain filter to balance data
domains. We evaluate the proposed method on the WMT 2018 Parallel Corpus
Filtering shared task, and on our own web-crawled Japanese-Chinese parallel
corpus. Our method significantly outperforms baselines and achieves a new
state-of-the-art. In an unsupervised setting, our method achieves comparable
performance to the top-1 supervised method. We also evaluate on a web-crawled
Japanese-Chinese parallel corpus that we make publicly available.

本文提出了一种利用预训练语言模型过滤爬取数据中的噪声句对的方法，并利用 BERT 的多语言能力度量语句的平行性，使用生成预训练（GPT）语言模型作为领域过滤器来平衡数据领域，通过在 WMT 2018 平行语料库过滤共享任务上的实验以及本文所提供的 Web-crawled 日译中平行语料库上的实验，证明该方法明显优于基准线，并取得了新的最新成果。