The introduction of ChatGPT has led to a significant increase in the
utilization of Large Language Models (LLMs) for addressing downstream tasks.
There's an increasing focus on cost-efficient training and deployment within
this context. Low-cost training and deployment of LLMs represent the future
development trend. This paper reviews the evolution of large language model
training techniques and inference deployment technologies aligned with this
emerging trend. The discussion on training includes various aspects, including
data preprocessing, training architecture, pre-training tasks, parallel
training, and relevant content related to model fine-tuning. On the inference
side, the paper covers topics such as model compression, parallel computation,
memory scheduling, and structural optimization. It also explores LLMs'
utilization and provides insights into their future development.

ChatGPT 的引入使得大型语言模型（LLM）被广泛应用于解决后续任务，而在此背景下，低成本的 LLM 培训和部署代表着未来的发展趋势。本文回顾了与这一新兴趋势相一致的大型语言模型培训技术和推理部署技术的演变。在培训方面，讨论了数据预处理、培训架构、预训练任务、并行培训以及与模型微调相关的内容。在推理方面，本文涵盖了模型压缩、并行计算、内存调度和结构优化等主题。它还探讨了 LLM 的利用并提供了对其未来发展的见解。