With increasing scale, large language models demonstrate both quantitative
improvement and new qualitative capabilities, especially as zero-shot learners,
like GPT-3. However, these results rely heavily on delicate prompt design and
large computation. In this work, we explore whether the strong zero-shot
ability could be achieved at a smaller model scale without any external
supervised data. To achieve this goal, we revisit masked language modeling and
present a geometry-guided self-supervised learning method (Go-tuningfor short)
by taking a small number of task-aware self-supervised data to update language
models further. Experiments show that Go-tuning can enable T5-small (80M)
competitive zero-shot results compared with large language models, such as
T5-XL (3B). We also apply Go-tuning on multi-task settings and develop a
multi-task model, mgo-T5 (250M). It can reach the average performance of OPT
(175B) on 9 datasets.

本文探讨如何在较小模型的规模下，通过几何引导的自监督学习方法进行任务感知的自监督数据调整，实现与大型语言模型相当的零样本能力。Go-tuning 方法的实验表明 T5-small（80M）能够达到 T5-XL（3B）的竞争零样本结果，并开发了一个多任务模型 mgo-T5（250M），达到九个数据集的 OPT（175B）的平均性能。