Although large language models have achieved impressive zero-shot ability,
the huge model size generally incurs high cost. Recently, semi-parametric
language models, which augment a smaller language model with an external
retriever, have demonstrated promising language modeling capabilities. However,
it remains unclear whether such semi-parametric language models can perform
competitively well as their fully-parametric counterparts on zero-shot
generalization to downstream tasks. In this work, we introduce $\text{Zemi}$, a
zero-shot semi-parametric language model. To our best knowledge, this is the
first semi-parametric language model that can demonstrate strong zero-shot
performance on a wide range of held-out unseen tasks. We train $\text{Zemi}$
with a novel semi-parametric multitask prompted training paradigm, which shows
significant improvement compared with the parametric multitask training as
proposed by T0. Specifically, we augment the multitask training and zero-shot
evaluation with retrieval from a large-scale task-agnostic unlabeled corpus. In
order to incorporate multiple potentially noisy retrieved augmentations, we
further propose a novel $\text{augmentation fusion}$ module leveraging
perceiver resampler and gated cross-attention. Notably, our proposed
$\text{Zemi}_\text{LARGE}$ outperforms T0-3B by 16% on all seven evaluation
tasks while being 3.9x smaller in model size.

这篇论文提出了一种名为 Zemi 的基于半参数模型的语言模型，采用了一种新型的半参数多任务提示训练范式，并引入了增强融合模块，显著提高了零点泛化能力，且模型规模比已有模型小 3.9 倍。