In clinical dictation, utterances after automatic speech recognition (ASR)
without explicit punctuation marks may lead to the misunderstanding of dictated
reports. To give a precise and understandable clinical report with ASR,
automatic punctuation restoration is required. Considering a practical
scenario, we propose a fast and light pre-trained model for Chinese medical
punctuation restoration based on 'pretraining and fine-tuning' paradigm. In
this work, we distill pre-trained models by incorporating supervised
contrastive learning and a novel auxiliary pre-training task (Punctuation Mark
Prediction) to make it well-suited for punctuation restoration. Our experiments
on various distilled models reveal that our model can achieve 95% performance
while 10% model size relative to state-of-the-art Chinese RoBERTa.

基于 “预训练和微调” 范式的快速轻量级中文医学标点还原模型，通过引入监督对比学习和新颖的辅助预训练任务（标点符号预测），蒸馏预训练模型以适应标点还原需求，实验表明，该模型在相对于最先进的中文 RoBERTa 模型的 10% 模型尺寸下能够达到 95% 的性能。