Pathology text mining is a challenging task given the reporting variability
and constant new findings in cancer sub-type definitions. However, successful
text mining of a large pathology database can play a critical role to advance
'big data' cancer research like similarity-based treatment selection, case
identification, prognostication, surveillance, clinical trial screening, risk
stratification, and many others. While there is a growing interest in
developing language models for more specific clinical domains, no
pathology-specific language space exist to support the rapid data-mining
development in pathology space. In literature, a few approaches fine-tuned
general transformer models on specialized corpora while maintaining the
original tokenizer, but in fields requiring specialized terminology, these
models often fail to perform adequately. We propose PathologyBERT - a
pre-trained masked language model which was trained on 347,173 histopathology
specimen reports and publicly released in the Huggingface repository. Our
comprehensive experiments demonstrate that pre-training of transformer model on
pathology corpora yields performance improvements on Natural Language
Understanding (NLU) and Breast Cancer Diagnose Classification when compared to
nonspecific language models.

介绍了 PathologyBERT - 一个采用 347,173 个组织病理学报告进行预训练的蒙面语言模型，以支持病理学领域的文本挖掘，并与其他模型进行了性能比较，以提高肿瘤识别和自然语言理解等方面的性能。