Large language models have emerged as a versatile tool but are challenging to
apply to tasks lacking large inference budgets and large in-domain training
sets. This work formalizes these constraints and distinguishes four important
variables: the pretraining budget (for training before the target domain is
known), the specialization budget (for training after the target domain is
known), the inference budget, and the in-domain training set size. Across these
settings, we compare different approaches from the machine learning literature.
Limited by inference cost, we find better alternatives to the standard practice
of training very large vanilla transformer models. In particular, we show that
hyper-networks and mixture of experts have better perplexity for large
pretraining budgets, while small models trained on importance sampled datasets
are attractive for large specialization budgets.

大语言模型在应用到缺乏大量推理预算和大量领域内训练集的任务中具有多样性，但具有挑战性。本研究规范了这些约束，并区分了四个重要变量：预训练预算（用于在目标领域之前进行训练）、专业预算（用于在目标领域之后进行训练）、推理预算和领域内训练集的大小。在这些设置中，我们比较了机器学习文献中的不同方法。受到推理成本的限制，我们找到了比训练非常大的基础变压器模型更好的替代方案。特别是，我们发现超网络和专家混合模型对于大的预训练预算具有更好的困惑度，而在重要抽样数据集上训练的小型模型对于大的专业预算是有吸引力的。