Recently, pretrained language models based on BERT have been introduced for
the French biomedical domain. Although these models have achieved
state-of-the-art results on biomedical and clinical NLP tasks, they are
constrained by a limited input sequence length of 512 tokens, which poses
challenges when applied to clinical notes. In this paper, we present a
comparative study of three adaptation strategies for long-sequence models,
leveraging the Longformer architecture. We conducted evaluations of these
models on 16 downstream tasks spanning both biomedical and clinical domains.
Our findings reveal that further pre-training an English clinical model with
French biomedical texts can outperform both converting a French biomedical BERT
to the Longformer architecture and pre-training a French biomedical Longformer
from scratch. The results underscore that long-sequence French biomedical
models improve performance across most downstream tasks regardless of sequence
length, but BERT based models remain the most efficient for named entity
recognition tasks.

比较了三种适用于长序列模型的适应策略，研究结果表明，用法文生物医学文本进一步预训练英文临床模型，优于将法文生物医学 BERT 转换为 Longformer 架构或从头预训练法文生物医学 Longformer，而对于命名实体识别任务，BERT 模型仍然是最高效的。