In this paper, we explore the utility of \textit{Translationese} as synthetic
data created using machine translation for pre-training language models (LMs).
Pre-training requires vast amounts of monolingual data, which is mostly
unavailable for languages other than English. Recently, there has been a
growing interest in using synthetic data to address this data scarcity. We take
the case of English and Indic languages and translate web-crawled monolingual
documents (clean) into the target language. Then, we train language models
containing 28M and 85M parameters on this translationese data (synthetic). We
show that their performance on downstream natural language understanding and
generative tasks is only 3.56\% poorer on NLU tasks and 1.51\% on NLG tasks
than LMs pre-trained on clean data. Further, we propose the use of lightweight
\textit{TinyLMs} pre-trained on clean data to filter synthetic data efficiently
which significantly improves the performance of our models. We also find that
LMs trained on synthetic data strongly benefit from extended pretraining on a
tiny fraction (10\%) of clean data. We release the data we collected and
created as a part of this work, \textit{IndicMonoDoc}, the largest collection
of monolingual document-level corpora, which we hope will help bridge the gap
between English and non-English performance for large language models.

通过使用机器翻译创建合成数据（Translationese）预训练语言模型（LMs），本研究探讨了将 Translationese 作为一种用于预训练语言模型的可用性。我们以英语和印度语为例，将网络抓取的单语文档翻译成目标语言，然后在这些合成数据（synthetic）上用含有 2800 万和 8500 万参数的语言模型进行训练。实验证明，在下游自然语言理解和生成任务中，与基于干净数据进行预训练的语言模型相比，我们的模型性能仅差 3.56％（自然语言理解任务）和 1.51％（自然语言生成任务）。此外，我们提出了使用基于干净数据进行预训练的轻量级 TinyLMs 有效过滤合成数据的方法，从而显著提高模型性能。我们还发现，在一个微小比例（10％）的干净数据上进行扩展预训练能显著改善预训练合成数据的语言模型。我们通过这项工作收集和创建的数据集 IndicMonoDoc 是最大的单语文档级语料库，希望能够弥补英语和非英语大型语言模型性能差距。