The integration of Large Language Models (LLMs) like GPT-4 into traditional
Natural Language Processing (NLP) tasks has opened new avenues for enhancing
model performance while reducing the reliance on extensive human annotations.
This paper presents a novel approach that leverages the Chain of Thought (CoT)
prompting technique to distill knowledge from GPT-4, subsequently applying it
to improve the efficiency and effectiveness of a smaller model, BERT, on Named
Entity Recognition (NER) tasks. Our method involves a two-phase training
process: initially employing GPT-4 annotated data for pre-training and then
refining the model with a combination of distilled and original human-annotated
data. The results demonstrate that our mixed-training strategy significantly
outperforms models trained solely on human annotations, achieving superior
F1-scores and showcasing a cost-effective solution for resource-limited or
closed-network settings. The study also discusses the challenges encountered,
such as LLM output variability and the tendency towards hallucinations,
proposing future work directions to enhance prompt design and annotation
selection. Our findings indicate a promising synergy between LLM insights and
traditional NLP techniques, paving the way for more accessible and robust NLP
applications.

该研究将大型语言模型（LLMs）如 GPT-4 与传统自然语言处理（NLP）任务相结合，利用思维链（CoT）提示技术从 GPT-4 中提取知识，并应用于改进较小模型 BERT 在命名实体识别（NER）任务中的效率和效果。通过采用两阶段训练过程，该方法在预训练阶段使用 GPT-4 标注数据，并结合蒸馏和原始人标注数据来完善模型。结果表明，我们的混合训练策略明显优于仅使用人工标注的模型，达到了更高的 F1 得分，并展示了在资源有限或封闭网络环境下的成本效益解决方案。研究还讨论了遇到的挑战，如 LLM 输出的可变性和偏向幻觉，提出了改进提示设计和注释选择的未来工作方向。我们的发现显示出 LLM 洞察力与传统 NLP 技术之间的有希望的协同作用，为更易于接近和强大的 NLP 应用铺平了道路。