In order to appropriately filter multi-modality data sets on a web-scale, it
becomes crucial to employ suitable filtering methods to boost performance and
reduce training costs. For instance, LAION papers employs the CLIP score filter
to select data with CLIP scores surpassing a certain threshold. On the other
hand, T-MARS achieves high-quality data filtering by detecting and masking text
within images and then filtering by CLIP score. Through analyzing the dataset,
we observe a significant proportion of redundant information, such as numbers,
present in the textual content. Our experiments on a subset of the data unveil
the profound impact of these redundant elements on the CLIP scores. A logical
approach would involve reevaluating the CLIP scores after eliminating these
influences. Experimentally, our text-based CLIP filter outperforms the
top-ranked method on the ``small scale" of DataComp (a data filtering
benchmark) on ImageNet distribution shifts, achieving a 3.6% performance
improvement. The results also demonstrate that our proposed text-masked filter
outperforms the original CLIP score filter when selecting the top 40% of the
data. The impact of numbers on CLIP and their handling provide valuable
insights for improving the effectiveness of CLIP training, including language
rewrite techniques.

通过分析多模态数据集，我们观察到包含数字等冗余信息对 CLIP 得分的深远影响，因此提出了一种基于文本的过滤方法，实验证明在选择数据的过程中超过原始 CLIP 得分过滤器的性能表现，并提供了改善 CLIP 训练有效性的有价值洞见，包括语言重写技术。

数字中的魔鬼：走向稳健的多模式数据过滤器

Devil in the Number: Towards Robust Multi-modality Data Filter

Almost all the state-of-the-art neural networks for computer vision tasks are
trained by (1) pre-training on a large-scale dataset and (2) finetuning on the
target dataset. This strategy helps reduce dependence on the target dataset and
improves convergence rate and generalization on the target task. Although
pre-training on large-scale datasets is very useful, its foremost disadvantage
is high training cost. To address this, we propose efficient filtering methods
to select relevant subsets from the pre-training dataset. Additionally, we
discover that lowering image resolutions in the pre-training step offers a
great trade-off between cost and performance. We validate our techniques by
pre-training on ImageNet in both the unsupervised and supervised settings and
finetuning on a diverse collection of target datasets and tasks. Our proposed
methods drastically reduce pre-training cost and provide strong performance
boosts. Finally, we improve standard ImageNet pre-training by 1-3% by tuning
available models on our subsets and pre-training on a dataset filtered from a
larger scale dataset.

提出了用于计算机视觉任务的预训练神经网络的有效过滤方法，包括图像分辨率下采样和数据子集筛选，验证结果表明这种方法明显降低了训练成本并提高了性能。

高效条件预训练用于迁移学习

Efficient Conditional Pre-training for Transfer Learning

This paper investigates the use of Machine Translation (MT) to bootstrap a
Natural Language Understanding (NLU) system for a new language for the use case
of a large-scale voice-controlled device. The goal is to decrease the cost and
time needed to get an annotated corpus for the new language, while still having
a large enough coverage of user requests. Different methods of filtering MT
data in order to keep utterances that improve NLU performance and
language-specific post-processing methods are investigated. These methods are
tested in a large-scale NLU task with translating around 10 millions training
utterances from English to German. The results show a large improvement for
using MT data over a grammar-based and over an in-house data collection
baseline, while reducing the manual effort greatly. Both filtering and
post-processing approaches improve results further.

本文研究了使用机器翻译（MT）为大规模语音控制设备的新语言引导自然语言理解（NLU）系统的使用情况，并探讨了不同的 MT 数据过滤方法以及语言特定的后处理方法在大规模 NLU 任务中的应用。结果表明，使用 MT 数据可以大大提高 NLU 的性能并减少人工工作量。