With the rise of large-scale pre-trained language models, open-domain
question-answering (ODQA) has become an important research topic in NLP. Based
on the popular pre-training fine-tuning approach, we posit that an additional
in-domain pre-training stage using a large-scale, natural, and diverse
question-answering (QA) dataset can be beneficial for ODQA. Consequently, we
propose a novel QA dataset based on the Common Crawl project in this paper.
Using the readily available schema.org annotation, we extract around 130
million multilingual question-answer pairs, including about 60 million English
data-points. With this previously unseen number of natural QA pairs, we
pre-train popular language models to show the potential of large-scale
in-domain pre-training for the task of question-answering. In our experiments,
we find that pre-training question-answering models on our Common Crawl
Question Answering dataset (CCQA) achieves promising results in zero-shot, low
resource and fine-tuned settings across multiple tasks, models and benchmarks.

本文提出一种在 Common Crawl 项目的基础上，使用大规模、自然、多样化问答数据集进行领域内预训练的方法，该方法可以用于 open-domain question-answering 任务中的零样本、低资源和微调设置，展示了预训练在该任务中的潜力。

CCQA：用于模型预训练的新型 Web 规模问答数据集

CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training

Distributed word representations, or word vectors, have recently been applied
to many tasks in natural language processing, leading to state-of-the-art
performance. A key ingredient to the successful application of these
representations is to train them on very large corpora, and use these
pre-trained models in downstream tasks. In this paper, we describe how we
trained such high quality word representations for 157 languages. We used two
sources of data to train these models: the free online encyclopedia Wikipedia
and data from the common crawl project. We also introduce three new word
analogy datasets to evaluate these word vectors, for French, Hindi and Polish.
Finally, we evaluate our pre-trained word vectors on 10 languages for which
evaluation datasets exists, showing very strong performance compared to
previous models.

本文介绍了一种方法，通过使用来自于维基百科和共同抓取项目的大型数据，并针对 157 种语言训练高质量的词向量。此外，我们还推出了三个新的单词类比数据集，以评估这些词向量。最后，我们针对 10 种有评估数据集存在的语言对我们的预训练词向量进行了评估，显示出与以前模型相比非常强的性能。