We introduce a new domain expert mixture (DEMix) layer that enables
conditioning a language model (LM) on the domain of the input text. A DEMix
layer is a collection of expert feedforward networks, each specialized to a
domain, that makes the LM modular: experts can be mixed, added or removed after
initial training. Extensive experiments with autoregressive transformer LMs (up
to 1.3B parameters) show that DEMix layers reduce test-time perplexity,
increase training efficiency, and enable rapid adaptation with little overhead.
We show that mixing experts during inference, using a parameter-free weighted
ensemble, allows the model to better generalize to heterogeneous or unseen
domains. We also show that experts can be added to iteratively incorporate new
domains without forgetting older ones, and that experts can be removed to
restrict access to unwanted domains, without additional training. Overall,
these results demonstrate benefits of explicitly conditioning on textual
domains during language modeling.

该研究介绍了一种新的域专家混合层 (DEMix)，能够使语言模型 (LM) 与输入文本的领域相适应。DEMix 层是由专家前馈网络组成的集合，每个专家网络专门处理一个领域，使得 LM 模块化：专家可以在初始训练后混合、添加或删除。该模型在自回归变压器 LM 上进行了广泛的实验 (高达 1.3B 参数)，结果表明 DEMix 层降低了测试时间内的困惑度，提高了训练效率，并允许快速适应而不会增加负担。同时显示，在推断时混合专家，使用无参数加权集成，使模型更好地推广到异构或未见领域。实验还表明，可以添加专家来逐步合并新领域，而不会遗忘旧领域，而且还可以删除专家以限制对不需要的领域的访问，而无需额外的训练。总之，这些结果表明，显式地依赖文本领域进行的语言建模具有益处。