We investigate the out-of-domain generalization of random feature (RF) models
and Transformers. We first prove that in the `generalization on the unseen
(GOTU)' setting, where training data is fully seen in some part of the domain
but testing is made on another part, and for RF models in the small feature
regime, the convergence takes place to interpolators of minimal degree as in
the Boolean case (Abbe et al., 2023). We then consider the sparse target regime
and explain how this regime relates to the small feature regime, but with a
different regularization term that can alter the picture in the non-Boolean
case. We show two different outcomes for the sparse regime with q-ary data
tokens: (1) if the data is embedded with roots of unities, then a min-degree
interpolator is learned like in the Boolean case for RF models, (2) if the data
is not embedded as such, e.g., simply as integers, then RF models and
Transformers may not learn minimal degree interpolators. This shows that the
Boolean setting and its roots of unities generalization are special cases where
the minimal degree interpolator offers a rare characterization of how learning
takes place. For more general integer and real-valued settings, a more nuanced
picture remains to be fully characterized.

研究无域推广的随机特征模型和 Transformer 在不同领域数据上的泛化能力，证明了小特征情形下的收敛性，并解释了稀疏目标情形与小特征情形的关系，同时展示了不同数据的情况下 RF 模型和 Transformer 学习能力的差异。