This paper addresses the growing need for efficient large language models
(LLMs) on mobile devices, driven by increasing cloud costs and latency
concerns. We focus on designing top-quality LLMs with fewer than a billion
parameters, a practical choice for mobile deployment. Contrary to prevailing
belief emphasizing the pivotal role of data and parameter quantity in
determining model quality, our investigation underscores the significance of
model architecture for sub-billion scale LLMs. Leveraging deep and thin
architectures, coupled with embedding sharing and grouped-query attention
mechanisms, we establish a strong baseline network denoted as MobileLLM, which
attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M
state-of-the-art models. Additionally, we propose an immediate block-wise
weight sharing approach with no increase in model size and only marginal
latency overhead. The resultant models, denoted as MobileLLM-LS, demonstrate a
further accuracy enhancement of 0.7%/0.8% than MobileLLM 125M/350M. Moreover,
MobileLLM model family shows significant improvements compared to previous
sub-billion models on chat benchmarks, and demonstrates close correctness to
LLaMA-v2 7B in API calling tasks, highlighting the capability of small models
for common on-device use cases.

通过设计深而瘦的体系结构以及嵌入共享和分组查询注意机制，我们提出了一种名为 MobileLLM 的强基线网络，它在先前的 125M/350M 最先进模型上分别获得 2.7%/4.3% 的准确度提升。此外，我们还提出了一种即时的分块权重共享方法，不增加模型大小且仅有微小的延迟开销。MobileLLM-LS 模型进一步提升了 0.7%/0.8% 的准确度，相较于 MobileLLM 125M/350M。此外，MobileLLM 模型系列在聊天基准测试中相较于之前的次十亿模型有显著提升，并在 API 调用任务中表现出接近 LLaMA-v2 7B 的正确性，突显了小型模型在常见设备使用情景中的能力。