We study the effectiveness of a simple approach to develop a small base
language model (LM) starting from an existing large base LM: first inherit a
few transformer blocks from the larger LM, and then train this smaller model on
a very small subset (0.1\%) of the raw pretraining data of the larger model. We
call our simple recipe Inheritune and first demonstrate it for building a small
base LM with 1.5B parameters using 1B tokens (and a starting few layers of
larger LM of 3B parameters); we do this using a single A6000 GPU for less than
half a day. Across 9 diverse evaluation datasets as well as the MMLU benchmark,
the resulting model compares favorably to publicly available base models of
1B-2B size, some of which have been trained using 50-1000 times more tokens.
We investigate Inheritune in a slightly different setting where we train
small LMs utilizing larger LMs and their full pre-training dataset. Here we
show that smaller LMs trained utilizing some of the layers of GPT2-medium
(355M) and GPT-2-large (770M) can effectively match the val loss of their
bigger counterparts when trained from scratch for the same number of training
steps on OpenWebText dataset with 9B tokens. We analyze our recipe with
extensive experiments and demonstrate it efficacy on diverse settings. Our code
is available at this https URL

我们研究了使用继承和训练的简单方法来构建小型基础语言模型的有效性，并通过实验证明其在各种设置中的功效。

用较少的令牌预训练小型基础语言模型

Pre-training Small Base LMs with Fewer Tokens

Large Language Models (LLMs) have shown promising performance in
knowledge-intensive reasoning tasks that require a compound understanding of
knowledge. However, deployment of the LLMs in real-world applications can be
challenging due to their high computational requirements and concerns on data
privacy. Previous studies have focused on building task-specific small language
models (LMs) by fine-tuning them with labeled data or distilling LLMs. However,
these approaches are ill-suited for knowledge-intensive reasoning tasks due to
the limited capacity of small LMs in memorizing the knowledge required.
Motivated by our theoretical analysis on memorization, we propose
Knowledge-Augmented Reasoning Distillation (KARD), a novel method that
fine-tunes small LMs to generate rationales with augmented knowledge retrieved
from an external knowledge base. Moreover, we further propose a neural reranker
to obtain documents relevant to rationale generation. We empirically show that
KARD significantly improves the performance of small T5 and Flan-T5 models on
the challenging knowledge-intensive reasoning datasets, namely MedQA-USMLE and
StrategyQA. Notably, our method makes the 250M models achieve superior
performance against the fine-tuned 3B models, having 12 times larger
parameters, on both MedQA-USMLE and StrategyQA benchmarks.

提出了知识增强推理提炼（KARD）这一新颖的方法，以从外部知识库检索的增强知识 fine-tune 小型 LM，来生成 rationale，并且进一步提出了神经重新排序器以获取与理性产生相关的文档。该方法在知识密集型推理数据集上显著提高了小型 T5 和 Flan-T5 模型的性能。