In-context learning (ICL) is a new learning paradigm that has gained
popularity along with the development of large language models. In this work,
we adapt a recently proposed hardness metric, pointwise $\mathcal{V}$-usable
information (PVI), to an in-context version (in-context PVI). Compared to the
original PVI, in-context PVI is more efficient in that it requires only a few
exemplars and does not require fine-tuning. We conducted a comprehensive
empirical analysis to evaluate the reliability of in-context PVI. Our findings
indicate that in-context PVI estimates exhibit similar characteristics to the
original PVI. Specific to the in-context setting, we show that in-context PVI
estimates remain consistent across different exemplar selections and numbers of
shots. The variance of in-context PVI estimates across different exemplar
selections is insignificant, which suggests that in-context PVI are stable.
Furthermore, we demonstrate how in-context PVI can be employed to identify
challenging instances. Our work highlights the potential of in-context PVI and
provides new insights into the capabilities of ICL.

通过将最近提出的硬度度量点点式 V - 可用信息（PVI）适应为上下文版本（上下文 PVI），我们评估了上下文 PVI 的可靠性，发现它与原始 PVI 具有类似的特征。此外，我们展示了如何利用上下文 PVI 来识别具有挑战性的实例，突显了上下文 PVI 的潜力并提供了对应于上下文学习能力的新见解。