Deploying Large Language Models (LLMs) on mobile devices makes all the
capabilities of natural language processing available on the device. An
important use case of LLMs is question answering, which can provide accurate
and contextually relevant answers to a wide array of user queries. We describe
how we managed to port state of the art LLMs to mobile devices, enabling them
to operate natively on the device. We employ the llama.cpp framework, a
flexible and self-contained C++ framework for LLM inference. We selected a
6-bit quantized version of the Orca-Mini-3B model with 3 billion parameters and
present the correct prompt format for this model. Experimental results show
that LLM inference runs in interactive speed on a Galaxy S21 smartphone and
that the model delivers high-quality answers to user queries related to
questions from different subjects like politics, geography or history.

将大型语言模型部署在移动设备上，使得所有自然语言处理的能力可在设备上使用；LLM 的重要用例是问答系统，可以提供准确和上下文相关的回答给用户的各种查询，并通过将 Orca-Mini-3B 模型的 6 位量化版本应用在 Galaxy S21 智能手机上，实验结果表明 LLM 推理在交互速度下进行，并能给出高质量的与政治、地理或历史相关的用户查询的答案。