Though discourse parsing can help multiple NLP fields, there has been no wide
language model search done on implicit discourse relation classification. This
hinders researchers from fully utilizing public-available models in discourse
analysis. This work is a straightforward, fine-tuned discourse performance
comparison of seven pre-trained language models. We use PDTB-3, a popular
discourse relation annotated dataset. Through our model search, we raise SOTA
to 0.671 ACC and obtain novel observations. Some are contrary to what has been
reported before (Shi and Demberg, 2019b), that sentence-level pre-training
objectives (NSP, SBO, SOP) generally fail to produce the best performing model
for implicit discourse relation classification. Counterintuitively,
similar-sized PLMs with MLM and full attention led to better performance.

基于 PDTB-3 数据集，通过对七个预训练语言模型的直接性能比较，我们的模型搜索将 SOTA 提高到了 0.671 的准确率，同时得到了一些新的观察结果，其中包括相对于之前的报道（Shi 和 Demberg, 2019b），句子级预训练目标（NSP，SBO，SOP）通常无法产生最佳性能的模型。反直觉的是，具有类似规模的 MLM 和全关注机制的 PLM 表现更好。