The scientific scale-up of large language models (LLMs) necessitates a
comprehensive understanding of their scaling properties. However, the existing
literature on the scaling properties only yields an incomplete answer:
optimization loss decreases predictably as the model size increases, in line
with established scaling law; yet no scaling law for task has been established
and the task performances are far from predictable during scaling. Task
performances typically show minor gains on small models until they improve
dramatically once models exceed a size threshold, exemplifying the ``emergent
abilities''. In this study, we discover that small models, although they
exhibit minor performance, demonstrate critical and consistent task performance
improvements that are not captured by conventional evaluation strategies due to
insufficient measurement resolution. To measure such improvements, we introduce
PassUntil, an evaluation strategy through massive sampling in the decoding
phase. We conduct quantitative investigations into the scaling law of task
performance. Firstly, a strict task scaling law is identified, enhancing the
predictability of task performances. Remarkably, we are able to predict the
performance of the 2.4B model on code generation with merely 0.05\% deviation
before training starts. Secondly, underpinned by PassUntil, we observe concrete
evidence of emergent abilities and ascertain that they are not in conflict with
the continuity of performance improvement. Their semblance to break-through is
that their scaling curve cannot be fitted by standard scaling law function. We
then introduce a mathematical definition for the emergent abilities. Through
the definition, we refute a prevalent ``multi-step reasoning hypothesis''
regarding the genesis of emergent abilities and propose a new hypothesis with a
satisfying fit to the observed scaling curve.

通过大规模采样在解码阶段引入 PassUntil 评估策略，本研究量化了任务性能的扩展规律并发现了突现能力的具体证据，从而推翻了有关突现能力产生的常见 “多步推理假设”，提出了一种符合观察到的扩展曲线的新假设。