Text-to-Table aims to generate structured tables to convey the key information from unstructured documents. Existing text-to-table datasets are typically oriented English, limiting the research in non-English languages. Meanwhile, the emergence of large language models (LLMs) has shown great success as general task solvers in multi-lingual settings (e.g., ChatGPT), theoretically enabling text-to-table in other languages. In this paper, we propose a Chinese text-to-table dataset, CT-Eval, to benchmark LLMs on this task. Our preliminary analysis of English text-to-table datasets highlights two key factors for dataset construction: data diversity and data hallucination. Inspired by this, the CT-Eval dataset selects a popular Chinese multidisciplinary online encyclopedia as the source and covers 28 domains to ensure data diversity. To minimize data hallucination, we first train an LLM to judge and filter out the task samples with hallucination, then employ human annotators to clean the hallucinations in the validation and testing sets. After this process, CT-Eval contains 88.6K task samples. Using CT-Eval, we evaluate the performance of open-source and closed-source LLMs. Our results reveal that zero-shot LLMs (including GPT-4) still have a significant performance gap compared with human judgment. Furthermore, after fine-tuning, open-source LLMs can significantly improve their text-to-table ability, outperforming GPT-4 by a large margin. In short, CT-Eval not only helps researchers evaluate and quickly understand the Chinese text-to-table ability of existing LLMs but also serves as a valuable resource to significantly improve the text-to-table performance of LLMs.

通过构建一个中文文本到表格的数据集 CT-Eval，并利用该数据集评估开源和闭源的大型语言模型(包括 GPT-4)的性能，研究结果显示零翻译大型语言模型仍然与人类判断存在明显的性能差距，但经过微调后，开源的大型语言模型的文本到表格能力可以显著提高，超过了 GPT-4 很大的程度。这表明 CT-Eval 不仅可以帮助研究人员评估和快速了解现有大型语言模型的中文文本到表格能力，还可作为极大提升大型语言模型文本到表格性能的宝贵资源。

CT-Eval: 大型语言模型中中国文本到表格性能的基准评估