In recent years, large language models (LMs) have achieved remarkable
progress across various natural language processing tasks. As pre-training and
fine-tuning are costly and might negatively impact model performance, it is
desired to efficiently adapt an existing model to different conditions such as
styles, sentiments or narratives, when facing different audiences or scenarios.
However, efficient adaptation of a language model to diverse conditions remains
an open challenge. This work is inspired by the observation that text
conditions are often associated with selection of certain words in a context.
Therefore we introduce LM-Switch, a theoretically grounded, lightweight and
simple method for generative language model conditioning. We begin by
investigating the effect of conditions in Hidden Markov Models (HMMs), and
establish a theoretical connection with language model. Our finding suggests
that condition shifts in HMMs are associated with linear transformations in
word embeddings. LM-Switch is then designed to deploy a learnable linear factor
in the word embedding space for language model conditioning. We show that
LM-Switch can model diverse tasks, and achieves comparable or better
performance compared with state-of-the-art baselines in LM detoxification and
generation control, despite requiring no more than 1% of parameters compared
with baselines and little extra time overhead compared with base LMs. It is
also able to learn from as few as a few sentences or one document. Moreover, a
learned LM-Switch can be transferred to other LMs of different sizes, achieving
a detoxification performance similar to the best baseline. We will make our
code available to the research community following publication.

本文介绍了一种理论上自洽、轻量、简单的 LM-Switch 语言模型调节方法，其通过在线性变换词嵌入空间来实现语言模型的高效适应，该方法只需要与基线相比不到 1% 的参数，且与基准 LM 相比，只需要很少的额外时间开销即可学习语言模型。