We analyze how well pre-trained large language models (e.g., Llama2, GPT-4,
Claude 3, etc) can do linear and non-linear regression when given in-context
examples, without any additional training or gradient updates. Our findings
reveal that several large language models (e.g., GPT-4, Claude 3) are able to
perform regression tasks with a performance rivaling (or even outperforming)
that of traditional supervised methods such as Random Forest, Bagging, or
Gradient Boosting. For example, on the challenging Friedman #2 regression
dataset, Claude 3 outperforms many supervised methods such as AdaBoost, SVM,
Random Forest, KNN, or Gradient Boosting. We then investigate how well the
performance of large language models scales with the number of in-context
exemplars. We borrow from the notion of regret from online learning and
empirically show that LLMs are capable of obtaining a sub-linear regret.

预训练的大型语言模型在给定上下文示例时，无需任何额外训练或梯度更新即可进行线性和非线性回归，发现多个大型语言模型在回归任务上的表现能与传统的监督方法如随机森林、装袋法或梯度提升相媲美甚至更好，此外，通过实证研究发现，大型语言模型的性能与上下文示例的数量具有亚线性的关系。