Learning quality document embeddings is a fundamental problem in natural
language processing (NLP), information retrieval (IR), recommendation systems,
and search engines. Despite recent advances in the development of
transformer-based models that produce sentence embeddings with self-contrastive
learning, the encoding of long documents (Ks of words) is still challenging
with respect to both efficiency and quality considerations. Therefore, we train
Longfomer-based document encoders using a state-of-the-art unsupervised
contrastive learning method (SimCSE). Further on, we complement the baseline
method -- siamese neural network -- with additional convex neural networks
based on functional Bregman divergence aiming to enhance the quality of the
output document representations. We show that overall the combination of a
self-contrastive siamese network and our proposed neural Bregman network
outperforms the baselines in two linear classification settings on three long
document topic classification tasks from the legal and biomedical domains.

本文提出了一种基于 Longfomer 的文档编码器，使用最新的无监督对比学习方法 SimCSE 进行训练，结合基于函数 Bregman 散度的凸神经网络以提高输出文档表示的质量，实验证明该方法在法律和生物医学领域的三个长文档主题分类任务中优于基线模型。