Large Language Models (LLMs) often do not perform well on queries that
require the aggregation of information across texts. To better evaluate this
setting and facilitate modeling efforts, we introduce TACT - Text And
Calculations through Tables, a dataset crafted to evaluate LLMs' reasoning and
computational abilities using complex instructions. TACT contains challenging
instructions that demand stitching information scattered across one or more
texts, and performing complex integration on this information to generate the
answer. We construct this dataset by leveraging an existing dataset of texts
and their associated tables. For each such tables, we formulate new queries,
and gather their respective answers. We demonstrate that all contemporary LLMs
perform poorly on this dataset, achieving an accuracy below 38\%. To pinpoint
the difficulties and thoroughly dissect the problem, we analyze model
performance across three components: table-generation, Pandas
command-generation, and execution. Unexpectedly, we discover that each
component presents substantial challenges for current LLMs. These insights lead
us to propose a focused modeling framework, which we refer to as IE as a tool.
Specifically, we propose to add "tools" for each of the above steps, and
implement each such tool with few-shot prompting. This approach shows an
improvement over existing prompting techniques, offering a promising direction
for enhancing model capabilities in these tasks.

使用 TACT 数据集评估了大型语言模型（LLMs）的推理和计算能力，发现现有模型在整合分散信息和执行复杂集成任务方面表现不佳。提出了一个名为 IE 作为工具的新建模型框架，通过为每个步骤添加工具并采用 few-shot prompting 方法，有效提升了模型在这些任务中的能力。