Large Language Models (LLMs), with their remarkable ability to tackle
challenging and unseen reasoning problems, hold immense potential for tabular
learning, that is vital for many real-world applications. In this paper, we
propose a novel in-context learning framework, FeatLLM, which employs LLMs as
feature engineers to produce an input data set that is optimally suited for
tabular predictions. The generated features are used to infer class likelihood
with a simple downstream machine learning model, such as linear regression and
yields high performance few-shot learning. The proposed FeatLLM framework only
uses this simple predictive model with the discovered features at inference
time. Compared to existing LLM-based approaches, FeatLLM eliminates the need to
send queries to the LLM for each sample at inference time. Moreover, it merely
requires API-level access to LLMs, and overcomes prompt size limitations. As
demonstrated across numerous tabular datasets from a wide range of domains,
FeatLLM generates high-quality rules, significantly (10% on average)
outperforming alternatives such as TabLLM and STUNT.

提出了一种新颖的上下文学习框架 FealtLLM，利用大型语言模型作为特征工程师，生成适合表格预测的优化输入数据集，在推断阶段使用生成的特征和简单的下游机器学习模型 (如线性回归) 来推断分类的可能性，从而实现高性能的小样本学习。FeatLLM 框架只使用简单的预测模型和推断时发现的特征，相较于现有的基于大型语言模型的方法，FeatLLM 可以消除每个样本都需要查询大型语言模型的需求和克服提示大小的限制，并且仅需要对大型语言模型的 API 进行访问。在多个来自不同领域的表格数据集上的实验证明，FeatLLM 生成高质量的规则，明显 (平均达到 10%) 优于 TabLLM 和 STUNT 等替代方法。

大型语言模型可自动为少样本表格学习生成特征

Large Language Models Can Automatically Engineer Features for Few-Shot  Tabular Learning

Recent development of deep neural networks (DNNs) for tabular learning has
largely benefited from the capability of DNNs for automatic feature
interaction. However, the heterogeneity nature of tabular features makes such
features relatively independent, and developing effective methods to promote
tabular feature interaction still remains an open problem. In this paper, we
propose a novel Graph Estimator, which automatically estimates the relations
among tabular features and builds graphs by assigning edges between related
features. Such relation graphs organize independent tabular features into a
kind of graph data such that interaction of nodes (tabular features) can be
conducted in an orderly fashion. Based on our proposed Graph Estimator, we
present a bespoke Transformer network tailored for tabular learning, called
T2G-Former, which processes tabular data by performing tabular feature
interaction guided by the relation graphs. A specific Cross-level Readout
collects salient features predicted by the layers in T2G-Former across
different levels, and attains global semantics for final prediction.
Comprehensive experiments show that our T2G-Former achieves superior
performance among DNNs and is competitive with non-deep Gradient Boosted
Decision Tree models.

本文提出了一种自动估计表格数据特征间关系的图估算器，根据所建立的关系图组织各独立特征并实现节点间的有序交互，进而进行表格学习。结合该方法，我们提出了一种基于 Transformer 网络的 T2G-Former，能够在不同层次上收集 T2G-Former 中的显著特征并加以综合，实现对表格数据进行有效处理和预测，进而取得了优异的表现。