Tabular data is ubiquitous in real-world applications and abundant on the
web, yet its annotation has traditionally required human labor, posing a
significant scalability bottleneck for tabular machine learning. Our
methodology can successfully annotate a large amount of tabular data and can be
flexibly steered to generate various types of annotations based on specific
research objectives, as we demonstrate with SQL annotation and input-target
column annotation as examples. As a result, we release AnnotatedTables, a
collection of 32,119 databases with LLM-generated annotations. The dataset
includes 405,616 valid SQL programs, making it the largest SQL dataset with
associated tabular data that supports query execution. To further demonstrate
the value of our methodology and dataset, we perform two follow-up research
studies. 1) We investigate whether LLMs can translate SQL programs to Rel
programs, a database language previously unknown to LLMs, while obtaining the
same execution results. Using our Incremental Prompt Engineering methods based
on execution feedback, we show that LLMs can produce adequate translations with
few-shot learning. 2) We evaluate the performance of TabPFN, a recent neural
tabular classifier trained on Bayesian priors, on 2,720 tables with
input-target columns identified and annotated by LLMs. On average, TabPFN
performs on par with the baseline AutoML method, though the relative
performance can vary significantly from one data table to another, making both
models viable for practical applications depending on the situation. Our
findings underscore the potential of LLMs in automating the annotation of large
volumes of diverse tabular data.

我们的研究展示了 LLM 在自动标注大量不同表格数据方面的潜力，通过 SQL 程序翻译、输入 - 目标列标注以及神经表格分类器的性能评估等研究，证明了 LLM 方法可以有效地自动标注大量表格数据并为各种具体的研究目标生成多样化的注释。

AnnotatedTables：一份带有语言模型注释的大型表格数据集

AnnotatedTables: A Large Tabular Dataset with Language Model Annotations

Data annotation is the labeling or tagging of raw data with relevant
information, essential for improving the efficacy of machine learning models.
The process, however, is labor-intensive and expensive. The emergence of
advanced Large Language Models (LLMs), exemplified by GPT-4, presents an
unprecedented opportunity to revolutionize and automate the intricate process
of data annotation. While existing surveys have extensively covered LLM
architecture, training, and general applications, this paper uniquely focuses
on their specific utility for data annotation. This survey contributes to three
core aspects: LLM-Based Data Annotation, Assessing LLM-generated Annotations,
and Learning with LLM-generated annotations. Furthermore, the paper includes an
in-depth taxonomy of methodologies employing LLMs for data annotation, a
comprehensive review of learning strategies for models incorporating
LLM-generated annotations, and a detailed discussion on primary challenges and
limitations associated with using LLMs for data annotation. As a key guide,
this survey aims to direct researchers and practitioners in exploring the
potential of the latest LLMs for data annotation, fostering future advancements
in this critical domain. We provide a comprehensive papers list at
https://github.com/Zhen-Tan-dmml/LLM4Annotation.git.

数据标注是提高机器学习模型效果的标记或标签化原始数据的过程，使用大型语言模型（LLMs）可以革新和自动化数据标注过程。本研究着重于 LLM 在数据标注中的特定用途，探讨了 LLM 基于数据标注、评估 LLM 生成的标注以及使用 LLM 生成的标注进行学习的方法。此外，还提供了 LLM 在数据标注中的方法学分类、包含 LLM 生成标注的模型的学习策略综述，并详细讨论了使用 LLM 进行数据标注所面临的主要挑战和限制。本文旨在为研究人员和从业者指导利用最新的 LLMs 进行数据标注，推动该关键领域的未来发展。