Compositional generalization--understanding unseen combinations of seen
primitives--is an essential reasoning capability in human intelligence. The AI
community mainly studies this capability by fine-tuning neural networks on lots
of training samples, while it is still unclear whether and how in-context
learning--the prevailing few-shot paradigm based on large language
models--exhibits compositional generalization. In this paper, we present CoFe,
a test suite to investigate in-context compositional generalization. We find
that the compositional generalization performance can be easily affected by the
selection of in-context examples, thus raising the research question what the
key factors are to make good in-context examples for compositional
generalization. We study three potential factors: similarity, diversity and
complexity. Our systematic experiments indicate that in-context examples should
be structurally similar to the test case, diverse from each other, and
individually simple. Furthermore, two strong limitations are observed:
in-context compositional generalization on fictional words is much weaker than
that on commonly used ones; it is still critical that the in-context examples
should cover required linguistic structures, even though the backbone model has
been pre-trained on large corpus. We hope our analysis would facilitate the
understanding and utilization of in-context learning paradigm.

本文介绍了 CoFe 测试套件，以研究上下文中符合规范的泛化，研究发现，应该具有与测试案例结构相似，彼此之间不同和个别简单的上下文实例，还应该覆盖必需的语言结构。