We introduce GLM-130B, a bilingual (English and Chinese) pre-trained language
model with 130 billion parameters. It is an attempt to open-source a 100B-scale
model at least as good as GPT-3 and unveil how models of such a scale can be
successfully pre-trained. Over the course of this effort, we face numerous
unexpected technical and engineering challenges, particularly on loss spikes
and disconvergence. In this paper, we introduce the training process of
GLM-130B including its design choices, training strategies for both efficiency
and stability, and engineering efforts. The resultant GLM-130B model offers
significant outperformance over GPT-3 175B on a wide range of popular English
benchmarks while the performance advantage is not observed in OPT-175B and
BLOOM-176B. It also consistently and significantly outperforms ERNIE TITAN 3.0
260B -- the largest Chinese language model -- across related benchmarks.
Finally, we leverage a unique scaling property of GLM-130B to reach INT4
quantization, without quantization aware training and with almost no
performance loss, making it the first among 100B-scale models. More
importantly, the property allows its effective inference on 4$\times$RTX 3090
(24G) or 8$\times$RTX 2080 Ti (11G) GPUs, the most ever affordable GPUs
required for using 100B-scale models. The GLM-130B model weights are publicly
accessible and its code, training logs, related toolkit, and lessons learned
are open-sourced at this https URL .

GLM-130B 是一种双语预训练语言模型，拥有 1300 亿个参数。通过本文我们介绍了 GLM-130B 的训练过程、设计选择、训练策略、工程努力、以及它有别于其他模型的独特的可扩展性等特点。我们观察到该模型在学习效率和稳定性方面都有非常好的表现，并能在英文和中文语料库中显著地超越 ERNIE TITAN 3.0 和 GPT-3 175B 等知名模型，且有别于其他模型的独特的可扩展性，可以在相对较低的成本下进行使用。