Scientific Information Extraction (ScientificIE) is a critical task that
involves the identification of scientific entities and their relationships. The
complexity of this task is compounded by the necessity for domain-specific
knowledge and the limited availability of annotated data. Two of the most
popular datasets for ScientificIE are SemEval-2018 Task-7 and SciERC. They have
overlapping samples and differ in their annotation schemes, which leads to
conflicts. In this study, we first introduced a novel approach based on
multi-task learning to address label variations. We then proposed a soft
labeling technique that converts inconsistent labels into probabilistic
distributions. The experimental results demonstrated that the proposed method
can enhance the model robustness to label noise and improve the end-to-end
performance in both ScientificIE tasks. The analysis revealed that label
variations can be particularly effective in handling ambiguous instances.
Furthermore, the richness of the information captured by label variations can
potentially reduce data size requirements. The findings highlight the
importance of releasing variation labels and promote future research on other
tasks in other domains. Overall, this study demonstrates the effectiveness of
multi-task learning and the potential of label variations to enhance the
performance of ScientificIE.

本研究通过多任务学习和标签变异的方法，提高科学信息抽取的性能，并展示了标签变异在处理模糊实例和减少数据需求方面的潜力。

通过多任务学习解决科学信息提取中的标签变异问题

Solving Label Variation in Scientific Information Extraction via  Multi-Task Learning

Our Visual Analytics (VA) tool ScrutinAI supports human analysts to
investigate interactively model performanceand data sets. Model performance
depends on labeling quality to a large extent. In particular in medical
settings, generation of high quality labels requires in depth expert knowledge
and is very costly. Often, data sets are labeled by collecting opinions of
groups of experts. We use our VA tool to analyse the influence of label
variations between different experts on the model performance. ScrutinAI
facilitates to perform a root cause analysis that distinguishes weaknesses of
deep neural network (DNN) models caused by varying or missing labeling quality
from true weaknesses. We scrutinize the overall detection of intracranial
hemorrhages and the more subtle differentiation between subtypes in a publicly
available data set.

我们的可视化分析工具 ScrutinAI 支持人工分析师与模型性能和数据集进行互动研究。通过使用我们的可视化工具，我们分析了不同专家之间标签变异对模型性能的影响，并能够区分来自标签质量的变化或缺失的深度神经网络模型的缺陷和真实缺陷，以便在公开可用的数据集中全面检测颅内出血和不同类型之间的细微差别。