Large Language Models (LLMs), particularly those similar to ChatGPT, have
significantly influenced the field of Natural Language Processing (NLP). While
these models excel in general language tasks, their performance in
domain-specific downstream tasks such as biomedical and clinical Named Entity
Recognition (NER), Relation Extraction (RE), and Medical Natural Language
Inference (NLI) is still evolving. In this context, our study investigates the
potential of instruction tuning for biomedical language processing, applying
this technique to two general LLMs of substantial scale. We present a
comprehensive, instruction-based model trained on a dataset that consists of
approximately $200,000$ instruction-focused samples. This dataset represents a
carefully curated compilation of existing data, meticulously adapted and
reformatted to align with the specific requirements of our instruction-based
tasks. This initiative represents an important step in utilising such models to
achieve results on par with specialised encoder-only models like BioBERT and
BioClinicalBERT for various classical biomedical NLP tasks. Our work includes
an analysis of the dataset's composition and its impact on model performance,
providing insights into the intricacies of instruction tuning. By sharing our
codes, models, and the distinctively assembled instruction-based dataset, we
seek to encourage ongoing research and development in this area.

利用指令调整（instruction tuning）技术，对两个规模庞大的通用语言模型进行研究，以期在生物医学自然语言处理任务上取得类似于 BioBERT 和 BioClinicalBERT 等特定编码器模型的结果，并提供了相应的代码、模型和基于指令的数据集。

探索生物医学语言处理中指令调整的有效性

Exploring the Effectiveness of Instruction Tuning in Biomedical Language  Processing

Using language models (LMs) pre-trained in a self-supervised setting on large
corpora and then fine-tuning for a downstream task has helped to deal with the
problem of limited label data for supervised learning tasks such as Named
Entity Recognition (NER). Recent research in biomedical language processing has
offered a number of biomedical LMs pre-trained using different methods and
techniques that advance results on many BioNLP tasks, including NER. However,
there is still a lack of a comprehensive comparison of pre-training approaches
that would work more optimally in the biomedical domain. This paper aims to
investigate different pre-training methods, such as pre-training the biomedical
LM from scratch and pre-training it in a continued fashion. We compare existing
methods with our proposed pre-training method of initializing weights for new
tokens by distilling existing weights from the BERT model inside the context
where the tokens were found. The method helps to speed up the pre-training
stage and improve performance on NER. In addition, we compare how masking rate,
corruption strategy, and masking strategies impact the performance of the
biomedical LM. Finally, using the insights from our experiments, we introduce a
new biomedical LM (BIOptimus), which is pre-trained using Curriculum Learning
(CL) and contextualized weight distillation method. Our model sets new states
of the art on several biomedical Named Entity Recognition (NER) tasks. We
release our code and all pre-trained models

本研究通过比较不同的预训练方法，包括从头开始预训练生物医学语言模型和在连续环境中预训练模型，并利用 BERT 模型内上下文中的现有权重提炼初始化新标记的权重，加快预训练阶段并提高命名实体识别性能；此外，我们还比较了掩码率、损坏策略和掩码策略对生物医学语言模型性能的影响；最终，我们通过课程学习和上下文化权重提炼方法提出了一种新的生物医学语言模型 (BIOptimus)，在多项生物医学命名实体识别任务中创造了新的技术水平