The rapid growth of scientific publications, particularly during the COVID-19
pandemic, emphasizes the need for tools to help researchers efficiently
comprehend the latest advancements. One essential part of understanding
scientific literature is research aspect classification, which categorizes
sentences in abstracts to Background, Purpose, Method, and Finding. In this
study, we investigate the impact of different datasets on model performance for
the crowd-annotated CODA-19 research aspect classification task. Specifically,
we explore the potential benefits of using the large, automatically curated
PubMed 200K RCT dataset and evaluate the effectiveness of large language models
(LLMs), such as LLaMA, GPT-3, ChatGPT, and GPT-4. Our results indicate that
using the PubMed 200K RCT dataset does not improve performance for the CODA-19
task. We also observe that while GPT-4 performs well, it does not outperform
the SciBERT model fine-tuned on the CODA-19 dataset, emphasizing the importance
of a dedicated and task-aligned datasets dataset for the target task. Our code
is available at this https URL

本文研究了不同数据集对居民群众标记的研究方面分类任务的模型性能的影响，探讨了使用大型自动筛选的 PubMed 200K RCT 数据集的潜在好处，并评估了大型语言模型，如 LLaMA，GPT-3，ChatGPT 和 GPT-4 的有效性。研究结果表明，使用 PubMed 200K RCT 数据集不会提高 CODA-19 任务的性能。同时，尽管 GPT-4 表现良好，但它没有超越在 CODA-19 数据集上优化的 SciBERT 模型，强调了专门针对目标任务的专用数据集的重要性。