The increasing size of language models raises great research interests in parameter-efficient fine-tuning (e.g. Adapter, LoRA and prompt tuning) that freezes the pre-trained model, and injects small-scale trainable parameters for multiple downstream tasks. To further enhance the efficiency of fine-tuning, we propose a framework that integrates LoRA and structured layer pruning. In addition, based on MIMIC-IV-Note, we create two deidentified medical report summarization datasets. Further, We validate the integrated framework on the proposed two datasets and two medical dialogue datasets. By tuning 0.6% parameters of the original model and pruning over 30% Transformer-layers, the framework can speed up 100% of the training phase and reduce 50% of GPU memory usage, while preserving over 92% generation qualities on free-text sequence-to-sequence tasks.

本文介绍了一个包含 LoRA 和结构化层剪枝的模型微调框架，用于对医学报告进行摘要提取，并且该框架可以通过微调少量参数和剪枝一部分 Transformer 层，加速了训练速度，减少 GPU 内存的使用，并且保持自由文本生成质量的同时，提高了模型的效率和效益。

基于层裁剪的医疗报告摘要和医学对话生成参数高效微调