We experiment with COVID-Twitter-BERT and RoBERTa models to identify
informative COVID-19 tweets. We further experiment with adversarial training to
make our models robust. The ensemble of COVID-Twitter-BERT and RoBERTa obtains
a F1-score of 0.9096 (on the positive class) on the test data of WNUT-2020 Task
2 and ranks 1st on the leaderboard. The ensemble of the models trained using
adversarial training also produces similar result.

使用 COVID-Twitter-BERT 和 RoBERTa 模型实验性地识别 COVID-19 推文，进一步实验了对抗性训练，使我们的模型更加稳健，模型集成达到了 WNUT-2020 任务 2 测试数据中的 0.9096 的 F1 得分，并在排行榜上排名第一，对抗性训练的模型集成也产生了类似的结果。

WNUT-2020 任务 2: 使用集成和对抗训练稳健地识别有价值的 COVID-19 推文的 NutCracker

NutCracker at WNUT-2020 Task 2: Robustly Identifying Informative  COVID-19 Tweets using Ensembling and Adversarial Training

In this work, we release COVID-Twitter-BERT (CT-BERT), a transformer-based
model, pretrained on a large corpus of Twitter messages on the topic of
COVID-19. Our model shows a 10-30% marginal improvement compared to its base
model, BERT-Large, on five different classification datasets. The largest
improvements are on the target domain. Pretrained transformer models, such as
CT-BERT, are trained on a specific target domain and can be used for a wide
variety of natural language processing tasks, including classification,
question-answering and chatbots. CT-BERT is optimised to be used on COVID-19
content, in particular social media posts from Twitter.

该论文介绍了基于 transformer 的 COVID-Twitter-BERT 模型，通过对大量 Twitter 上有关 COVID-19 的消息进行预训练，该模型在五个不同的分类数据集上相较于 BERT-Large 基础模型提高了 10-30％，特别是在 COVID-19 内容（尤其是来自 Twitter 的社交媒体帖子）的分类中表现更突出，从而实现了优化。该模型可以用于文本分类、问答和聊天机器人等自然语言处理任务。