Pretrained language models have been used in various natural language
processing applications. In the mental health domain, domain-specific language
models are pretrained and released, which facilitates the early detection of
mental health conditions. Social posts, e.g., on Reddit, are usually long
documents. However, there are no domain-specific pretrained models for
long-sequence modeling in the mental health domain. This paper conducts
domain-specific continued pretraining to capture the long context for mental
health. Specifically, we train and release MentalXLNet and MentalLongformer
based on XLNet and Longformer. We evaluate the mental health classification
performance and the long-range ability of these two domain-specific pretrained
models. Our models are released in HuggingFace.

本文针对心理健康领域的 Reddit 帖子的长序列建模问题，进行了领域特定的持续预训练，基于 XLNet 和 Longformer 训练了 MentalXLNet 和 MentalLongformer 模型，评估了这两个模型的心理健康分类性能和长距离能力，并在 HuggingFace 上发布了这两个模型。