Pre-trained language models (PLMs) have achieved remarkable success in NLP tasks. Despite the great success, mainstream solutions largely follow the pre-training then finetuning paradigm, which brings in both high deployment costs and low training efficiency. Nevertheless, fine-tuning on a specific task is essential because PLMs are only pre-trained with language signal from large raw data. In this paper, we propose a novel fine-tuning-free strategy for language models, to consider both language signal and teacher signal. Teacher signal is an abstraction of a battery of downstream tasks, provided in a unified proposition format. Trained with both language and strong task-aware teacher signals in an interactive manner, our FreeLM model demonstrates strong generalization and robustness. FreeLM outperforms large models e.g., GPT-3 and InstructGPT, on a range of language understanding tasks in experiments. FreeLM is much smaller with 0.3B parameters, compared to 175B in these models.

本文提出了一种新颖的无微调的自然语言处理模型Fine-tuning-free strategy，通过使用语言和强任务感知的teacher signal进行交互式训练，提高了该模型在多项任务中的泛化性和鲁棒性，并且相对于大型模型如GPT-3和InstructGPT而言，该模型较小，只有0.3B的参数。

FreeLM：无微调语言模型