The ability of the foundation models heavily relies on large-scale, diverse,
and high-quality pretraining data. In order to improve data quality,
researchers and practitioners often have to manually curate datasets from
difference sources and develop dedicated data cleansing pipeline for each data
repository. Lacking a unified data processing framework, this process is
repetitive and cumbersome. To mitigate this issue, we propose a data processing
framework that integrates a Processing Module which consists of a series of
operators at different granularity levels, and an Analyzing Module which
supports probing and evaluation of the refined data. The proposed framework is
easy to use and highly flexible. In this demo paper, we first introduce how to
use this framework with some example use cases and then demonstrate its
effectiveness in improving the data quality with an automated evaluation with
ChatGPT and an end-to-end evaluation in pretraining the GPT-2 model. The code
and demonstration videos are accessible on GitHub.

基于统一数据处理框架，提出基于大规模、多样化和高质量预训练数据的基础模型，并通过例子和测试展示其在提高数据质量方面的效果。

预训练基础模型的整合数据处理框架

An Integrated Data Processing Framework for Pretraining Foundation  Models

We present Pathway, a new unified data processing framework that can run
workloads on both bounded and unbounded data streams. The framework was created
with the original motivation of resolving challenges faced when analyzing and
processing data from the physical economy, including streams of data generated
by IoT and enterprise systems. These required rapid reaction while calling for
the application of advanced computation paradigms (machinelearning-powered
analytics, contextual analysis, and other elements of complex event
processing). Pathway is equipped with a Table API tailored for Python and
Python/SQL workflows, and is powered by a distributed incremental dataflow in
Rust. We describe the system and present benchmarking results which demonstrate
its capabilities in both batch and streaming contexts, where it is able to
surpass state-of-the-art industry frameworks in both scenarios. We also discuss
streaming use cases handled by Pathway which cannot be easily resolved with
state-of-the-art industry frameworks, such as streaming iterative graph
algorithms (PageRank, etc.).

我们提出了 Pathway，这是一个新的统一数据处理框架，可以在有界和无界数据流上运行工作负载。该框架旨在解决分析和处理物理经济数据时面临的挑战，包括物联网和企业系统生成的数据流。我们描述了该系统，并展示了其在批处理和流处理环境中的能力，超过了业界最先进的框架。我们还讨论了 Pathway 处理的流处理用例，这些用例无法轻松解决使用业界最先进的框架，例如流迭代图算法（PageRank 等）。