The availability of both structured and unstructured databases, such as
electronic health data, social media data, patent data, and surveys that are
often updated in real time, among others, has grown rapidly over the past
decade. With this expansion, the statistical and methodological questions
around data integration, or rather merging multiple data sources, has also
grown. Specifically, the science of the ``data cleaning pipeline'' contains
four stages that allow an analyst to perform downstream tasks, predictive
analyses, or statistical analyses on ``cleaned data.'' This article provides a
review of this emerging field, introducing technical terminology and commonly
used methods.

通过介绍技术术语和常用方法，综述了数据集成和数据清洗流程中的四个阶段，探讨了在现有的结构化和非结构化数据库中进行预测分析或统计分析所涉及的统计和方法学问题。

数据清洗流程初探

A Primer on the Data Cleaning Pipeline

The advancements of neural dialogue generation models show promising results
on modeling short-text conversations. However, training such models usually
needs a large-scale high-quality dialogue corpus, which is hard to access. In
this paper, we present a large-scale cleaned Chinese conversation dataset,
LCCC, which contains a base version (6.8million dialogues) and a large version
(12.0 million dialogues). The quality of our dataset is ensured by a rigorous
data cleaning pipeline, which is built based on a set of rules and a classifier
that is trained on manually annotated 110K dialogue pairs. We also release
pre-training dialogue models which are trained on LCCC-base and LCCC-large
respectively. The cleaned dataset and the pre-training models will facilitate
the research of short-text conversation modeling. All the models and datasets
are available at this https URL

本文介绍了一个大型的、经过清洗的中文对话数据集 LCCC，包含基础版和大型版两种版本，共计 680 万和 1200 万对话。数据集的质量通过一套规则和分类器确保。此外，本文还释放了 LCCC-base 和 LCCC-large 的预训练对话模型，这些数据集和模型将有助于研究短文本对话建模。