Large Language Models (LLMs) have demonstrated their exceptional performance in various complex code generation tasks. However, their broader adoption is limited by significant computational demands and high resource requirements, particularly memory and processing power. To mitigate such requirements, model pruning techniques are used to create more compact models with significantly fewer parameters. However, current approaches do not focus on the efficient extraction of programming-language-specific sub-models. In this work, we explore the idea of efficiently deriving coding-specific sub-models through unstructured pruning (i.e., Wanda). We investigate the impact of different domain-specific calibration datasets on pruning outcomes across three distinct domains and extend our analysis to extracting four language-specific sub-models: Python, Java, C++, and JavaScript. We are the first to efficiently extract programming-language-specific sub-models using appropriate calibration datasets while maintaining acceptable accuracy w.r.t. full models. We are also the first to provide analytical evidence that domain-specific tasks activate distinct regions within LLMs, supporting the creation of specialized sub-models through unstructured pruning. We believe that this work has significant potential to enhance LLM accessibility for coding by reducing computational requirements to enable local execution on consumer-grade hardware, and supporting faster inference times critical for real-time development feedback.

本研究解决了大型语言模型（LLMs）在编程任务中由于计算需求和资源要求高而限制其广泛应用的问题。文章提出了一种高效推导编程语言特定子模型的新方法，通过无结构修剪技术，并在多个领域测试不同的校准数据集对修剪效果的影响。研究表明，使用适当的校准数据集可以高效提取特定编程语言的子模型，同时保持与完整模型的可接受准确性，从而显著降低计算需求，增强LLMs在编程中的可及性。

通过资源高效修剪从大型语言模型中推导特定编码子模型