Large language models (LLMs) have achieved remarkable progress in various
natural language processing tasks with emergent abilities. However, they face
inherent limitations, such as an inability to access up-to-date information,
utilize external tools, or perform precise mathematical reasoning. In this
paper, we introduce Chameleon, a plug-and-play compositional reasoning
framework that augments LLMs to help address these challenges. Chameleon
synthesizes programs to compose various tools, including LLM models,
off-the-shelf vision models, web search engines, Python functions, and
rule-based modules tailored to user interests. Built on top of an LLM as a
natural language planner, Chameleon infers the appropriate sequence of tools to
compose and execute in order to generate a final response. We showcase the
adaptability and effectiveness of Chameleon on two tasks: ScienceQA and TabMWP.
Notably, Chameleon with GPT-4 achieves an 86.54% accuracy on ScienceQA,
significantly improving upon the best published few-shot model by 11.37%; using
GPT-4 as the underlying LLM, Chameleon achieves a 17.8% increase over the
state-of-the-art model, leading to a 98.78% overall accuracy on TabMWP. Further
studies suggest that using GPT-4 as a planner exhibits more consistent and
rational tool selection and is able to infer potential constraints given the
instructions, compared to other LLMs like ChatGPT.

本文提出了 Chameleon 框架，该框架通过在 Large language models（LLMs）上加上合成程序的方式来提高 LLMs 的能力，特别是在处理需要精确数学推理、使用外部工具或获取最新信息的任务时。作者通过使用科学问题答案（ScienceQA）和网页表格解析（TabMWP）两个任务来展示 Chameleon 的效率和适应性，结果表明 Chameleon 可以在这些任务上显著提高模型的准确率，并且使用 GPT-4 作为规划器可以更加稳定和合理地选择工具，并推断给定指令的潜在约束。