This paper presents an overview of the PromptCBLUE shared task
(this http URL) held in the CHIP-2023 Conference. This
shared task reformualtes the CBLUE benchmark, and provide a good testbed for
Chinese open-domain or medical-domain large language models (LLMs) in general
medical natural language processing. Two different tracks are held: (a) prompt
tuning track, investigating the multitask prompt tuning of LLMs, (b) probing
the in-context learning capabilities of open-sourced LLMs. Many teams from both
the industry and academia participated in the shared tasks, and the top teams
achieved amazing test results. This paper describes the tasks, the datasets,
evaluation metrics, and the top systems for both tasks. Finally, the paper
summarizes the techniques and results of the evaluation of the various
approaches explored by the participating teams.

该研究论文介绍了 PromptCBLUE 共享任务，该任务重塑了 CBLUE 基准，并为中文开放领域或医学领域的大型语言模型（LLMs）提供了一个良好的测试平台，探索了 LLMs 的多任务提示优化和开源 LLMs 的上下文学习能力调查，并总结了各参与团队在不同方法上的技术和结果评估。

CHIP2023 中 PromptCBLUE 共享任务概述

Overview of the PromptCBLUE Shared Task in CHIP2023

Biomedical language understanding benchmarks are the driving forces for
artificial intelligence applications with large language model (LLM) back-ends.
However, most current benchmarks: (a) are limited to English which makes it
challenging to replicate many of the successes in English for other languages,
or (b) focus on knowledge probing of LLMs and neglect to evaluate how LLMs
apply these knowledge to perform on a wide range of bio-medical tasks, or (c)
have become a publicly available corpus and are leaked to LLMs during
pre-training. To facilitate the research in medical LLMs, we re-build the
Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark into a
large scale prompt-tuning benchmark, PromptCBLUE. Our benchmark is a suitable
test-bed and an online platform for evaluating Chinese LLMs' multi-task
capabilities on a wide range bio-medical tasks including medical entity
recognition, medical text classification, medical natural language inference,
medical dialogue understanding and medical content/dialogue generation. To
establish evaluation on these tasks, we have experimented and report the
results with the current 9 Chinese LLMs fine-tuned with differtent fine-tuning
techniques.

为了促进医学大语言模型的研究，我们将中文生物医学语言理解评估 (CBLUE) 基准重新构建为一个大规模的提示调整基准，PromptCBLUE，用于评估中文大语言模型在广泛的生物医学任务上的多任务能力，包括医学实体识别、医学文本分类、医学自然语言推理、医学对话理解和医学内容 / 对话生成。在这些任务上，我们已经进行了实验并报告了目前使用不同微调技术微调的 9 个中文大语言模型的结果。