Text-to-Table aims to generate structured tables to convey the key
information from unstructured documents. Existing text-to-table datasets are
typically oriented English, limiting the research in non-English languages.
Meanwhile, the emergence of large language models (LLMs) has shown great
success as general task solvers in multi-lingual settings (e.g., ChatGPT),
theoretically enabling text-to-table in other languages. In this paper, we
propose a Chinese text-to-table dataset, CT-Eval, to benchmark LLMs on this
task. Our preliminary analysis of English text-to-table datasets highlights two
key factors for dataset construction: data diversity and data hallucination.
Inspired by this, the CT-Eval dataset selects a popular Chinese
multidisciplinary online encyclopedia as the source and covers 28 domains to
ensure data diversity. To minimize data hallucination, we first train an LLM to
judge and filter out the task samples with hallucination, then employ human
annotators to clean the hallucinations in the validation and testing sets.
After this process, CT-Eval contains 88.6K task samples. Using CT-Eval, we
evaluate the performance of open-source and closed-source LLMs. Our results
reveal that zero-shot LLMs (including GPT-4) still have a significant
performance gap compared with human judgment. Furthermore, after fine-tuning,
open-source LLMs can significantly improve their text-to-table ability,
outperforming GPT-4 by a large margin. In short, CT-Eval not only helps
researchers evaluate and quickly understand the Chinese text-to-table ability
of existing LLMs but also serves as a valuable resource to significantly
improve the text-to-table performance of LLMs.

通过构建一个中文文本到表格的数据集 CT-Eval，并利用该数据集评估开源和闭源的大型语言模型 (包括 GPT-4) 的性能，研究结果显示零翻译大型语言模型仍然与人类判断存在明显的性能差距，但经过微调后，开源的大型语言模型的文本到表格能力可以显著提高，超过了 GPT-4 很大的程度。这表明 CT-Eval 不仅可以帮助研究人员评估和快速了解现有大型语言模型的中文文本到表格能力，还可作为极大提升大型语言模型文本到表格性能的宝贵资源。

CT-Eval: 大型语言模型中中国文本到表格性能的基准评估

CT-Eval: Benchmarking Chinese Text-to-Table Performance in Large  Language Models

We study a new problem setting of information extraction (IE), referred to as
text-to-table. In text-to-table, given a text, one creates a table or several
tables expressing the main content of the text, while the model is learned from
text-table pair data. The problem setting differs from those of the existing
methods for IE. First, the extraction can be carried out from long texts to
large tables with complex structures. Second, the extraction is entirely
data-driven, and there is no need to explicitly define the schemas. As far as
we know, there has been no previous work that studies the problem. In this
work, we formalize text-to-table as a sequence-to-sequence (seq2seq) problem.
We first employ a seq2seq model fine-tuned from a pre-trained language model to
perform the task. We also develop a new method within the seq2seq approach,
exploiting two additional techniques in table generation: table constraint and
table relation embeddings. We consider text-to-table as an inverse problem of
the well-studied table-to-text, and make use of four existing table-to-text
datasets in our experiments on text-to-table. Experimental results show that
the vanilla seq2seq model can outperform the baseline methods of using relation
extraction and named entity extraction. The results also show that our method
can further boost the performances of the vanilla seq2seq model. We further
discuss the main challenges of the proposed task. The code and data are
available at this https URL

本文研究了信息提取的新问题 -- 文本到表格，通过学习文本 - 表格配对数据，训练模型以生成主要内容的表格。我们将其形式化为序列到序列问题，使用预训练语言模型 fine-tuned 后的 seq2seq 模型完成任务，并通过引入表格约束和表格关系嵌入来改进性能。实验结果显示，我们的方法可以进一步提高基线模型和关系提取 / 命名实体提取等传统方法的性能，论文还讨论了相关挑战。