In-context learning (ICL) is a new paradigm for natural language processing
that utilizes Generative Pre-trained Transformer (GPT)-like models. This
approach uses prompts that include in-context demonstrations to generate the
corresponding output for a new query input. However, applying ICL in real cases
does not scale with the number of samples, and lacks robustness to different
prompt templates and demonstration permutations. In this paper, we first show
that GPT-like models using ICL result in unreliable predictions based on a new
metric based on Shannon entropy. Then, to solve this problem, we propose a new
technique called the Linear Probe Calibration (LinC), a method that calibrates
the model's output probabilities, resulting in reliable predictions and
improved performance, while requiring only minimal additional samples (as few
as five labeled data samples). LinC significantly enhances the ICL test
performance of GPT models on various benchmark datasets, with an average
improvement of up to 21%, and up to a 50% improvement in some cases, and
significantly boosts the performance of PEFT methods, especially in the low
resource regime. Moreover, LinC achieves lower expected calibration error, and
is highly robust to varying label proportions, prompt templates, and
demonstration permutations. Our code is available at
https://github.com/mominabbass/LinC.

通过使用新的线性校准技术（LinC），在仅需少量数据样本（如仅五个标记数据样本）的情况下，对于以 in-context learning（ICL）为基础的 GPT 模型进行概率输出校准，从而达到可靠的预测和改善性能，并显著提升 GPT 模型在各种基准数据集上的测试性能，平均改善率高达 21％，某些情况下高达 50％，尤其在资源有限的情况下提升 PEFT 方法的性能，同时具有较低的期望校准误差，并对不同的标签比例、提示模板和演示排列都具有高度的鲁棒性。