In AI research, the optimization of Large Language Models (LLMs) remains a
significant challenge, crucial for advancing the field's practical applications
and sustainability. Building upon the foundational work of Professor Song Han's
lab at MIT, this paper introduces a novel approach in developing Mini-GPTs via
contextual pruning. Our methodology strategically prunes the computational
architecture of traditional LLMs, like Phi-1.5, focusing on retaining core
functionalities while drastically reducing model sizes. We employ the technique
across diverse and complex datasets, including US law, Medical Q&A, Skyrim
dialogue, English-Taiwanese translation, and Economics articles. The results
underscore the efficiency and effectiveness of contextual pruning, not merely
as a theoretical concept but as a practical tool in developing domain-specific,
resource-efficient LLMs. Contextual pruning is a promising method for building
domain-specific LLMs, and this research is a building block towards future
development with more hardware compute, refined fine-tuning, and quantization.

利用上下文剪枝技术开发 Mini-GPT 来优化大型语言模型（LLMs），通过对传统 LLMs 的计算架构进行剪枝，保留核心功能同时大幅减小模型大小，此方法在多样且复杂的数据集上展现了高效性和有效性，作为构建领域特定 LLMs 的潜力方法，是未来发展的基石。