Emergence, broadly conceptualized as the ``intelligent'' behaviors of LLMs,
has recently been studied and proved challenging to quantify due to the lack of
a measurable definition. Most commonly, it has been estimated statistically
through model performances across extensive datasets and tasks, which consumes
significant resources. In addition, such estimation is difficult to interpret
and may not accurately reflect the models' intrinsic emergence. In this work,
we propose a quantifiable solution for estimating emergence. Inspired by
emergentism in dynamics, we quantify the strength of emergence by comparing the
entropy reduction of the macroscopic (semantic) level with that of the
microscopic (token) level, both of which are derived from the representations
within the transformer block. Using a low-cost estimator, our quantification
method demonstrates consistent behaviors across a suite of LMs (GPT-2, GEMMA,
etc.) under both in-context learning (ICL) and natural sentences. Empirical
results show that (1) our method gives consistent measurements which align with
existing observations based on performance metrics, validating the
effectiveness of our emergence quantification; (2) our proposed metric uncovers
novel emergence patterns such as the correlations between the variance of our
metric and the number of ``shots'' in ICL, which further suggests a new way of
interpreting hallucinations in LLMs; (3) we offer a potential solution towards
estimating the emergence of larger and closed-resource LMs via smaller LMs like
GPT-2. Our codes are available at:
this https URL

我们提出了一种可量化的解决方案来估计新现象，在动力学中受到新兴主义启发，通过将转换器块中的表示计算的宏观（语义）级别和微观（令牌）级别的熵减进行比较，以量化新现象的强度。

大型语言模型中的出现量化

Quantifying Emergence in Large Language Models

We conducted more than 1.3 million comparisons of iris patterns encoded from
images collected at two Nigerian universities, which constitute the newly
available African Human Iris (AFHIRIS) database. The purpose was to discover
whether ethnic differences in iris structure and appearance such as the
textural feature size, as contrasted with an all-Chinese image database or an
American database in which only 1.53% were of African-American heritage, made a
material difference for iris discrimination. We measured a reduction in entropy
for the AFHIRIS database due to the coarser iris features created by the thick
anterior layer of melanocytes, and we found stochastic parameters that
accurately model the relevant empirical distributions. Quantile-Quantile
analysis revealed that a very small change in operational decision thresholds
for the African database would compensate for the reduced entropy and generate
the same performance in terms of resistance to False Matches. We conclude that
despite demographic difference, individuality can be robustly discerned by
comparison of iris patterns in this West African population.

尽管民族差异存在，可通过比较这个西非人群的虹膜模式来稳定地辨别个体特征。