We analyze how well pre-trained large language models (e.g., Llama2, GPT-4,
Claude 3, etc) can do linear and non-linear regression when given in-context
examples, without any additional training or gradient updates. Our findings
reveal that several large language models (e.g., GPT-4, Claude 3) are able to
perform regression tasks with a performance rivaling (or even outperforming)
that of traditional supervised methods such as Random Forest, Bagging, or
Gradient Boosting. For example, on the challenging Friedman #2 regression
dataset, Claude 3 outperforms many supervised methods such as AdaBoost, SVM,
Random Forest, KNN, or Gradient Boosting. We then investigate how well the
performance of large language models scales with the number of in-context
exemplars. We borrow from the notion of regret from online learning and
empirically show that LLMs are capable of obtaining a sub-linear regret.

预训练的大型语言模型在给定上下文示例时，无需任何额外训练或梯度更新即可进行线性和非线性回归，发现多个大型语言模型在回归任务上的表现能与传统的监督方法如随机森林、装袋法或梯度提升相媲美甚至更好，此外，通过实证研究发现，大型语言模型的性能与上下文示例的数量具有亚线性的关系。

从词语到数字：在上下文示例中，你的大型语言模型暗自成为一种有能力的回归器

From Words to Numbers: Your Large Language Model Is Secretly A Capable  Regressor When Given In-Context Examples

Prompt-based methods with large pre-trained language models (PLMs) have shown
impressive unaided performance across many NLP tasks. These models improve even
further with the addition of a few labeled in-context exemplars to guide output
generation. However, for more complex tasks such as dialogue state tracking
(DST), designing prompts that reliably convey the desired intent is nontrivial,
leading to unstable results. Furthermore, building in-context exemplars for
dialogue tasks is difficult because conversational contexts are long while
model input lengths are relatively short. To overcome these issues we first
adapt a meta-learning scheme to the dialogue domain which stabilizes the
ability of the model to perform well under various prompts. We additionally
design a novel training method to improve upon vanilla retrieval mechanisms to
find ideal in-context examples. Finally, we introduce a saliency model to limit
dialogue text length, allowing us to include more exemplars per query. In
effect, we are able to achieve highly competitive results for few-shot DST on
MultiWOZ.

通过元学习在对话领域稳定模型，设计新的训练方法改进检索机制寻找理想的示例，在句子长度有限的情况下，使用显著性模型限制对话文本长度，从而在 MultiWOZ 上实现了具有竞争力的少样本对话状态跟踪结果。