Automated knowledge curation for biomedical ontologies is key to ensure that
they remain comprehensive, high-quality and up-to-date. In the era of
foundational language models, this study compares and analyzes three NLP
paradigms for curation tasks: in-context learning (ICL), fine-tuning (FT), and
supervised learning (ML). Using the Chemical Entities of Biological Interest
(ChEBI) database as a model ontology, three curation tasks were devised. For
ICL, three prompting strategies were employed with GPT-4, GPT-3.5, BioGPT.
PubmedBERT was chosen for the FT paradigm. For ML, six embedding models were
utilized for training Random Forest and Long-Short Term Memory models. Five
setups were designed to assess ML and FT model performance across different
data availability scenarios.Datasets for curation tasks included: task 1
(620,386), task 2 (611,430), and task 3 (617,381), maintaining a 50:50 positive
versus negative ratio. For ICL models, GPT-4 achieved best accuracy scores of
0.916, 0.766 and 0.874 for tasks 1-3 respectively. In a direct comparison, ML
(trained on ~260,000 triples) outperformed ICL in accuracy across all tasks.
(accuracy differences: +.11, +.22 and +.17). Fine-tuned PubmedBERT performed
similarly to leading ML models in tasks 1 & 2 (F1 differences: -.014 and
+.002), but worse in task 3 (-.048). Simulations revealed performance declines
in both ML and FT models with smaller and higher imbalanced training data.
where ICL (particularly GPT-4) excelled in tasks 1 & 3. GPT-4 excelled in tasks
1 and 3 with less than 6,000 triples, surpassing ML/FT. ICL underperformed
ML/FT in task 2.ICL-augmented foundation models can be good assistants for
knowledge curation with correct prompting, however, not making ML and FT
paradigms obsolete. The latter two require task-specific data to beat ICL. In
such cases, ML relies on small pretrained embeddings, minimizing computational
demands.

自动化知识整理对于生物医学本体论至关重要，本研究比较和分析了三种自然语言处理范式（ICL、FT 和 ML）在整理任务中的效果，结果显示在特定数据情景下，ICL 表现出色，而 ML 和 FT 则需要任务特定的数据来超越 ICL。

基于上下文学习、微调和监督学习的生物医学知识整理的基准测试与分析：以生物学相关化学实体为例的一项专项研究

Benchmarking and Analyzing In-context Learning, Fine-tuning and  Supervised Learning for Biomedical Knowledge Curation: a focused study on  chemical entities of biological interest