The ability of large language models (LLMs) to "learn in context" based on
the provided prompt has led to an explosive growth in their use, culminating in
the proliferation of AI assistants such as ChatGPT, Claude, and Bard. These AI
assistants are known to be robust to minor prompt modifications, mostly due to
alignment techniques that use human feedback. In contrast, the underlying
pre-trained LLMs they use as a backbone are known to be brittle in this
respect. Building high-quality backbone models remains a core challenge, and a
common approach to assessing their quality is to conduct few-shot evaluation.
Such evaluation is notorious for being highly sensitive to minor prompt
modifications, as well as the choice of specific in-context examples. Prior
work has examined how modifying different elements of the prompt can affect
model performance. However, these earlier studies tended to concentrate on a
limited number of specific prompt attributes and often produced contradictory
results. Additionally, previous research either focused on models with fewer
than 15 billion parameters or exclusively examined black-box models like GPT-3
or PaLM, making replication challenging. In the present study, we decompose the
entire prompt into four components: task description, demonstration inputs,
labels, and inline instructions provided for each demonstration. We investigate
the effects of structural and semantic corruptions of these elements on model
performance. We study models ranging from 1.5B to 70B in size, using ten
datasets covering classification and generation tasks. We find that repeating
text within the prompt boosts model performance, and bigger models ($\geq$30B)
are more sensitive to the semantics of the prompt. Finally, we observe that
adding task and inline instructions to the demonstrations enhances model
performance even when the instructions are semantically corrupted.

通过对大语言模型进行随机组成的实验，研究了任务描述、示例输入、标签、行内指令等多个因素对模型性能的影响，发现重复文本、大型模型以及任务和行内指令对模型性能提升具有积极影响。

反演上下文学习：通过破坏理解提示

Deconstructing In-Context Learning: Understanding Prompts via Corruption

Large Transformer models pre-trained on massive unlabeled molecular data have
shown great success in predicting molecular properties. However, these models
can be prone to overfitting during fine-tuning, resulting in over-confident
predictions on test data that fall outside of the training distribution. To
address this issue, uncertainty quantification (UQ) methods can be used to
improve the models' calibration of predictions. Although many UQ approaches
exist, not all of them lead to improved performance. While some studies have
used UQ to improve molecular pre-trained models, the process of selecting
suitable backbone and UQ methods for reliable molecular uncertainty estimation
remains underexplored. To address this gap, we present MUBen, which evaluates
different combinations of backbone and UQ models to quantify their performance
for both property prediction and uncertainty estimation. By fine-tuning various
backbone molecular representation models using different molecular descriptors
as inputs with UQ methods from different categories, we critically assess the
influence of architectural decisions and training strategies. Our study offers
insights for selecting UQ and backbone models, which can facilitate research on
uncertainty-critical applications in fields such as materials science and drug
discovery.

本文介绍了如何使用不确定性量化（UQ）方法来改善预测分子属性的大型 Transformer 模型的置信度，并评估了不同的 backbone 和 UQ 模型组合的性能，提供了关于选择 UQ 和 backbone 模型的见解。