The meaning of complex phrases in natural language is composed of their individual components. The task of compositional generalization evaluates a model's ability to understand new combinations of components. Previous studies trained smaller, task-specific models, which exhibited poor generalization. While large language models (LLMs) exhibit impressive generalization abilities on many tasks through in-context learning (ICL), their potential for compositional generalization remains unexplored. In this paper, we first empirically investigate prevailing ICL methods in compositional generalization. We find that they struggle with complex compositional questions due to cumulative errors in long reasoning steps and intricate logic required for tool-making. Consequently, we propose a human-guided tool manipulation framework (HTM) that generates tools for sub-questions and integrates multiple tools. Our method enhances the effectiveness of tool creation and usage with minimal human effort. Experiments show that our method achieves state-of-the-art performance on two compositional generalization benchmarks and outperforms existing methods on the most challenging test split by 70%.

我们通过对现有ICL方法的实证调查发现其在复合泛化问题上表现不佳，因此我们提出了一种人工引导的工具操作框架(HTM)，它通过生成子问题的工具并整合多个工具来提高工具的创建和使用效果。实验证明，我们的方法在两个复合泛化基准测试上取得了最先进的性能，并在最具挑战性的测试中优于现有方法70%。

LLM能否找到绿色圆圈？构成泛化的调查和人类引导工具操作