We describe Bioformer team's participation in the multi-label topic
classification task for COVID-19 literature (track 5 of BioCreative VII). Topic
classification is performed using different BERT models (BioBERT, PubMedBERT,
and Bioformer). We formulate the topic classification task as a sentence pair
classification problem, where the title is the first sentence, and the abstract
is the second sentence. Our results show that Bioformer outperforms BioBERT and
PubMedBERT in this task. Compared to the baseline results, our best model
increased micro, macro, and instance-based F1 score by 8.8%, 15.5%, 7.4%,
respectively. Bioformer achieved the highest micro F1 and macro F1 scores in
this challenge. In post-challenge experiments, we found that pretraining of
Bioformer on COVID-19 articles further improves the performance.

本文介绍 Bioformer 团队在 BioCreative VII 的 track 5 中多标签主题分类任务中所采用的不同 BERT 模型 (BioBERT、PubMedBERT 和 Bioformer) 对 COVID-19 文献主题分类的表现。结果表明 Bioformer 在此任务中的表现优于 BioBERT 和 PubMedBERT，并使用 COVID-19 文章的预训练进一步提高了性能。

使用 Bioformer 进行 COVID-19 文献的多标记主题分类

Multi-label topic classification for COVID-19 literature with Bioformer

Biomedical question-answering (QA) has gained increased attention for its
capability to provide users with high-quality information from a vast
scientific literature. Although an increasing number of biomedical QA datasets
has been recently made available, those resources are still rather limited and
expensive to produce. Transfer learning via pre-trained language models (LMs)
has been shown as a promising approach to leverage existing general-purpose
knowledge. However, finetuning these large models can be costly and time
consuming, often yielding limited benefits when adapting to specific themes of
specialised domains, such as the COVID-19 literature. To bootstrap further
their domain adaptation, we propose a simple yet unexplored approach, which we
call biomedical entity-aware masking (BEM). We encourage masked language models
to learn entity-centric knowledge based on the pivotal entities characterizing
the domain at hand, and employ those entities to drive the LM fine-tuning. The
resulting strategy is a downstream process applicable to a wide variety of
masked LMs, not requiring additional memory or components in the neural
architectures. Experimental results show performance on par with
state-of-the-art models on several biomedical QA datasets.

本文提出了一种称为生物医学实体感知蒙版 (BEM) 的简单方法，以帮助受限于特定领域及其 COVID-19 文献的预训练语言模型进行微调，从而提升其生物医学问答数据集的性能。实验证明，该方法与先进模型具有相同的性能。

利用实体感知屏蔽策略提高低资源生物医学问答性能

Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies

We created this EVIDENCEMINER system for automatic textual evidence mining in
COVID-19 literature. EVIDENCEMINER is a web-based system that lets users query
a natural language statement and automatically retrieves textual evidence from
a background corpora for life sciences. It is constructed in a completely
automated way without any human effort for training data annotation.
EVIDENCEMINER is supported by novel data-driven methods for distantly
supervised named entity recognition and open information extraction. The named
entities and meta-patterns are pre-computed and indexed offline to support fast
online evidence retrieval. The annotation results are also highlighted in the
original document for better visualization. EVIDENCEMINER also includes
analytic functionalities such as the most frequent entity and relation
summarization.

本文介绍了一种基于数据驱动方法的自动文本证据挖掘系统 EVIDENCEMINER，用于 COVID-19 文献，可以查询自然语言语句并自动从生命科学的背景文献中检索文本证据，并具有快速检索、在线读取、高亮显示和分析等功能。