Social media platforms play an essential role in crisis communication, but
analyzing crisis-related social media texts is challenging due to their
informal nature. Transformer-based pre-trained models like BERT and RoBERTa
have shown success in various NLP tasks, but they are not tailored for
crisis-related texts. Furthermore, general-purpose sentence encoders are used
to generate sentence embeddings, regardless of the textual complexities in
crisis-related texts. Advances in applications like text classification,
semantic search, and clustering contribute to effective processing of
crisis-related texts, which is essential for emergency responders to gain a
comprehensive view of a crisis event, whether historical or real-time. To
address these gaps in crisis informatics literature, this study introduces
CrisisTransformers, an ensemble of pre-trained language models and sentence
encoders trained on an extensive corpus of over 15 billion word tokens from
tweets associated with more than 30 crisis events, including disease outbreaks,
natural disasters, conflicts, and other critical incidents. We evaluate
existing models and CrisisTransformers on 18 crisis-specific public datasets.
Our pre-trained models outperform strong baselines across all datasets in
classification tasks, and our best-performing sentence encoder improves the
state-of-the-art by 17.43% in sentence encoding tasks. Additionally, we
investigate the impact of model initialization on convergence and evaluate the
significance of domain-specific models in generating semantically meaningful
sentence embeddings. All models are publicly released
(this https URL), with the anticipation that they
will serve as a robust baseline for tasks involving the analysis of
crisis-related social media texts.

社交媒体平台在危机传播中起着重要作用，但由于其非正式性质，分析危机相关的社交媒体文本具有挑战性。为了解决危机信息学文献中的这些差距，本研究引入了 CrisisTransformers，这是一个预训练语言模型和句子编码器的集合，它们经过广泛的语料库训练，包含了超过 150 亿词元的推文，涉及 30 多个危机事件，包括疾病爆发、自然灾害、冲突和其他重大事件。评估现有模型和 CrisisTransformers 在 18 个危机特定的公共数据集上表现，我们的预训练模型在所有数据集的分类任务中都优于强基线，而我们表现最佳的句子编码器在句子编码任务中将现有技术水平提高了 17.43%。此外，我们研究了模型初始化对收敛性的影响，并评估了领域专用模型在生成语义有意义的句子嵌入中的重要性。所有模型已公开发布，以期成为分析危机相关社交媒体文本任务的强大基准。