Pretrained large generative language models have shown great performance on
many tasks, but exhibit low compositional generalization abilities. Scaling
such models has been shown to improve their performance on various NLP tasks
even just by conditioning them on a few examples to solve the task without any
fine-tuning (also known as in-context learning). In this work, we look at the
gap between the in-distribution (ID) and out-of-distribution (OOD) performance
of such models in semantic parsing tasks with in-context learning. In the ID
settings, the demonstrations are from the same split (test or train) that the
model is being evaluated on, and in the OOD settings, they are from the other
split. We look at how the relative generalization gap of in-context learning
evolves as models are scaled up. We evaluate four model families, OPT, BLOOM,
CodeGen and Codex on three semantic parsing datasets, CFQ, SCAN and GeoQuery
with different number of exemplars, and observe a trend of decreasing relative
generalization gap as models are scaled up.

本研究探讨了预训练大型生成语言模型在语义解析任务中在上下文学习时的分布内和分布外表现差异以及模型规模的影响，结果显示随着模型规模的增加，相对泛化差距逐渐减小。