In this paper, we introduce SciGen, a new challenge dataset for the task of
reasoning-aware data-to-text generation consisting of tables from scientific
articles and their corresponding descriptions. Describing scientific tables
goes beyond the surface realization of the table content and requires reasoning
over table values. The unique properties of SciGen are that (1) tables mostly
contain numerical values, and (2) the corresponding descriptions require
arithmetic reasoning. SciGen is therefore the first dataset that assesses the
arithmetic reasoning capabilities of generation models on complex input
structures, i.e., tables from scientific articles. We study the effectiveness
of state-of-the-art data-to-text generation models on SciGen and evaluate the
results using common metrics as well as human evaluation. Our results and
analyses show that (a) while humans like to reason for describing scientific
tables, the ability of state-of-the-art models is severely limited on this
task, (b) while adding more training data improves the results, it is not the
solution for reasoning-aware text generation, and (c) one of the main
bottlenecks for this task is the lack of proper automatic evaluation metrics.
The data, code, and annotations for human evaluation will be available at
this https URL SciGen opens new avenues for future research
in reasoning-aware text generation and evaluation.

本研究引入了 SciGen，一个考虑推理的数据到文本生成任务的数据集。SciGen 是第一个评估生成模型在复杂输入结构下算术推理能力的数据集，并表明表格描述需要考虑表格值的推理。在 SciGen 上进行实验表明，目前数据到文本生成模型的推理能力受限，缺乏适当的自动评估度量。