Foundational models with billions of parameters which have been trained on
large corpora of data have demonstrated non-trivial skills in a variety of
domains. However, due to their monolithic structure, it is challenging and
expensive to augment them or impart new skills. On the other hand, due to their
adaptation abilities, several new instances of these models are being trained
towards new domains and tasks. In this work, we study the problem of efficient
and practical composition of existing foundation models with more specific
models to enable newer capabilities. To this end, we propose CALM --
Composition to Augment Language Models -- which introduces cross-attention
between models to compose their representations and enable new capabilities.
Salient features of CALM are: (i) Scales up LLMs on new tasks by 're-using'
existing LLMs along with a few additional parameters and data, (ii) Existing
model weights are kept intact, and hence preserves existing capabilities, and
(iii) Applies to diverse domains and settings. We illustrate that augmenting
PaLM2-S with a smaller model trained on low-resource languages results in an
absolute improvement of up to 13\% on tasks like translation into English and
arithmetic reasoning for low-resource languages. Similarly, when PaLM2-S is
augmented with a code-specific model, we see a relative improvement of 40\%
over the base model for code generation and explanation tasks -- on-par with
fully fine-tuned counterparts.

通过提出 CALM，即 Composition to Augment Language Models，我们研究了现有基础模型与特定模型的有效和实用组合，以赋予其新的能力。CALM 通过引入模型之间的交叉关注来组合它们的表示，并实现新的功能。在实验证明，将 PaLM2-S 与在资源稀缺语言上训练的较小模型相结合，可以在英语翻译和低资源语言的算术推理等任务中带来最高 13％的绝对改进；同样，当 PaLM2-S 与特定于代码的模型相结合时，在代码生成和解释任务上与完全微调的模型相比，性能提升了 40％。