Large Language Models (LLMs) have demonstrated significant potential in transforming clinical applications. In this study, we investigate the efficacy of four techniques in adapting LLMs for clinical use-cases: continuous pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning dataset of 500 million tokens. Our evaluation across various clinical tasks reveals the impact of each technique. While continuous pretraining beyond 250 billion tokens yields marginal improvements on its own, it establishes a strong foundation for instruct fine-tuning. Notably, NEFTune, designed primarily to enhance generation quality, surprisingly demonstrates additional gains on our benchmark. Complex prompt engineering methods further enhance performance. These findings show the importance of tailoring fine-tuning strategies and exploring innovative techniques to optimize LLM performance in the clinical domain.

本研究解决了将大型语言模型应用于临床的有效性问题，探讨了持续预训练、指令微调、NEFTune和提示工程四种技术的适应性。研究发现，持续预训练虽然在超过2500亿标记时改善幅度有限，但为指令微调奠定了坚实基础，而NEFTune则意外地在基准测试中表现出额外提升，推动了临床领域模型性能的优化。

超越微调：释放临床大型语言模型的持续预训练潜力