"Bigger the better" has been the predominant trend in recent Large Language
Models (LLMs) development. However, LLMs do not suit well for scenarios that
require on-device processing, energy efficiency, low memory footprint, and
response efficiency. These requisites are crucial for privacy, security, and
sustainable deployment. This paper explores the "less is more" paradigm by
addressing the challenge of designing accurate yet efficient Small Language
Models (SLMs) for resource constrained devices. Our primary contribution is the
introduction of an accurate and fully transparent open-source 0.5 billion
(0.5B) parameter SLM, named MobiLlama, catering to the specific needs of
resource-constrained computing with an emphasis on enhanced performance with
reduced resource demands. MobiLlama is a SLM design that initiates from a
larger model and applies a careful parameter sharing scheme to reduce both the
pre-training and the deployment cost. Our work strives to not only bridge the
gap in open-source SLMs but also ensures full transparency, where complete
training data pipeline, training code, model weights, and over 300 checkpoints
along with evaluation codes is available at :
this https URL

通过引入准确而高效的开源 0.5 亿参数的小语言模型 MobiLlama，本文探讨了在资源受限设备上设计准确但高效的小语言模型的挑战，并关注性能提升和资源需求降低，以满足隐私、安全和可持续性部署的需求。

MobiLlama：面向准确轻量级全透明的 GPT

MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT

Implicit neural representation (INR) can describe the target scenes with high
fidelity using a small number of parameters, and is emerging as a promising
data compression technique. However, limited spectrum coverage is intrinsic to
INR, and it is non-trivial to remove redundancy in diverse complex data
effectively. Preliminary studies can only exploit either global or local
correlation in the target data and thus of limited performance. In this paper,
we propose a Tree-structured Implicit Neural Compression (TINC) to conduct
compact representation for local regions and extract the shared features of
these local representations in a hierarchical manner. Specifically, we use
Multi-Layer Perceptrons (MLPs) to fit the partitioned local regions, and these
MLPs are organized in tree structure to share parameters according to the
spatial distance. The parameter sharing scheme not only ensures the continuity
between adjacent regions, but also jointly removes the local and non-local
redundancy. Extensive experiments show that TINC improves the compression
fidelity of INR, and has shown impressive compression capabilities over
commercial tools and other deep learning based methods. Besides, the approach
is of high flexibility and can be tailored for different data and parameter
settings. The source code can be found at this https URL .

本文提出了一种基于树形结构的隐式神经网络压缩技术（TINC），利用多层感知机（MLPs）来适应局部区域，组织成树形结构来共享参数，以提高数据压缩效率并消除局部和非局部冗余。实验表明 TINC 在 INR 数据压缩方面具有很高的灵活性和可扩展性，并比商业工具和其他深度学习方法表现更好。