Prior-data fitted networks (PFNs) were recently proposed as a new paradigm
for machine learning. Instead of training the network to an observed training
set, a fixed model is pre-trained offline on small, simulated training sets
from a variety of tasks. The pre-trained model is then used to infer class
probabilities in-context on fresh training sets with arbitrary size and
distribution. Empirically, PFNs achieve state-of-the-art performance on tasks
with similar size to the ones used in pre-training. Surprisingly, their
accuracy further improves when passed larger data sets during inference. This
article establishes a theoretical foundation for PFNs and illuminates the
statistical mechanisms governing their behavior. While PFNs are motivated by
Bayesian ideas, a purely frequentistic interpretation of PFNs as pre-tuned, but
untrained predictors explains their behavior. A predictor's variance vanishes
if its sensitivity to individual training samples does and the bias vanishes
only if it is appropriately localized around the test feature. The transformer
architecture used in current PFN implementations ensures only the former. These
findings shall prove useful for designing architectures with favorable
empirical behavior.

该研究论文探讨了先验数据拟合网络（PFN）的理论基础，以及控制其行为的统计机制。虽然 PFN 是受贝叶斯思想启发的，但其行为可以纯粹地以预调整但未经训练的预测器来解释。