Training giant models from scratch for each complex task is resource- and
data-inefficient. To help develop models that can leverage existing systems, we
propose a new challenge: Learning to solve complex tasks by communicating with
existing agents (or models) in natural language. We design a synthetic
benchmark, CommaQA, with three complex reasoning tasks (explicit, implicit,
numeric) designed to be solved by communicating with existing QA agents. For
instance, using text and table QA agents to answer questions such as "Who had
the longest javelin throw from USA?". We show that black-box models struggle to
learn this task from scratch (accuracy under 50\%) even with access to each
agent's knowledge and gold facts supervision. In contrast, models that learn to
communicate with agents outperform black-box models, reaching scores of 100\%
when given gold decomposition supervision. However, we show that the challenge
of learning to solve complex tasks by communicating with existing agents
\emph{without relying on any auxiliary supervision or data} still remains
highly elusive. We release CommaQA, along with a compositional generalization
test split, to advance research in this direction. Dataset and Code available
at this https URL

为了解决模型训练的数据和资源浪费问题，本文提出了一项新的挑战 —— 通过自然语言与现有智能体通信以完成复杂任务。通过设计一个综合性的基准测试 ——CommaQA，该基准测试包括三项复杂的推理任务，旨在通过与现有的 QA 智能体进行沟通来解决这些任务。研究表明，传统的黑盒模型在使用现有智能体的知识和金标事实监督时也难以从头开始实现此任务。相比之下，学会与代理通信的模型表现更好，即使没有任何辅助监督和数据，这种通过与现有代理完成复杂任务的学习仍然是非常困难的。本文提供了 CommaQA 基准测试和组合泛化测试集，以推动这方面的研究。