The rapid advancement of large language models (LLMs) has led to significant
improvements in natural language processing but also poses challenges due to
their high computational and energy demands. This paper introduces a series of
research efforts focused on Super Tiny Language Models (STLMs), which aim to
deliver high performance with significantly reduced parameter counts. We
explore innovative techniques such as byte-level tokenization with a pooling
mechanism, weight tying, and efficient training strategies. These methods
collectively reduce the parameter count by $90\%$ to $95\%$ compared to
traditional models while maintaining competitive performance. This series of
papers will explore into various subproblems, including tokenizer-free models,
self-play based training, and alternative training objectives, targeting models
with 10M, 50M, and 100M parameters. Our ultimate goal is to make
high-performance language models more accessible and practical for a wide range
of applications.

通过引入一系列研究工作，本文介绍了超小型语言模型 (STLMs) 的创新技术和高性能表现，包括字节级的分词和汇聚机制、参数联系以及高效的训练策略，以实现与传统模型相比参数数量减少了 90% 至 95% 的竞争性表现，未来的研究将探索包括无分词模型、基于自我博弈的训练以及替代训练目标等多个子问题，旨在使高性能语言模型在更广泛的应用领域中更具可访问性和实用性。