Large Language Models (LLMs) demonstrate remarkable performance on a variety
of Natural Language Understanding (NLU) tasks, primarily due to their
in-context learning ability. This ability is utilized in our proposed
"CoThought" pipeline, which efficiently trains smaller "baby" language models
(BabyLMs) by leveraging the Chain of Thought (CoT) prompting of LLMs. Our
pipeline restructures a dataset of less than 100M in size using GPT-3.5-turbo,
transforming it into task-oriented, human-readable texts that are comparable to
the school texts for language learners. The BabyLM is then pretrained on this
restructured dataset in a RoBERTa (Liu et al., 2019) fashion. In evaluations
across 4 benchmarks, our BabyLM outperforms the RoBERTa-base in 10 linguistic,
NLU, and question answering tasks by more than 3 points, showing superior
ability to extract contextual information. These results suggest that compact
LMs pretrained on small, LLM-restructured data can better understand tasks and
achieve improved performance. The code for data processing and model training
is available at: this https URL

通过使用 CoThought 流水线，我们可以有效地训练较小的 BabyLM 语言模型，从而利用 LLMs 的上下文学习能力，将小于 100M 的数据集转变为适用于语言学习者的任务导向的可读文本，经 RoBERTa 的预训练后，BabyLM 在多项语言学、NLU 和问答任务中表现优越，超过 RoBERTa-base 的性能 3 个点以上，显示出更好的上下文信息提取能力。