Large Language Models (LLMs), with their remarkable ability to tackle
challenging and unseen reasoning problems, hold immense potential for tabular
learning, that is vital for many real-world applications. In this paper, we
propose a novel in-context learning framework, FeatLLM, which employs LLMs as
feature engineers to produce an input data set that is optimally suited for
tabular predictions. The generated features are used to infer class likelihood
with a simple downstream machine learning model, such as linear regression and
yields high performance few-shot learning. The proposed FeatLLM framework only
uses this simple predictive model with the discovered features at inference
time. Compared to existing LLM-based approaches, FeatLLM eliminates the need to
send queries to the LLM for each sample at inference time. Moreover, it merely
requires API-level access to LLMs, and overcomes prompt size limitations. As
demonstrated across numerous tabular datasets from a wide range of domains,
FeatLLM generates high-quality rules, significantly (10% on average)
outperforming alternatives such as TabLLM and STUNT.

提出了一种新颖的上下文学习框架 FealtLLM，利用大型语言模型作为特征工程师，生成适合表格预测的优化输入数据集，在推断阶段使用生成的特征和简单的下游机器学习模型 (如线性回归) 来推断分类的可能性，从而实现高性能的小样本学习。FeatLLM 框架只使用简单的预测模型和推断时发现的特征，相较于现有的基于大型语言模型的方法，FeatLLM 可以消除每个样本都需要查询大型语言模型的需求和克服提示大小的限制，并且仅需要对大型语言模型的 API 进行访问。在多个来自不同领域的表格数据集上的实验证明，FeatLLM 生成高质量的规则，明显 (平均达到 10%) 优于 TabLLM 和 STUNT 等替代方法。