In recent years, contrastive learning (CL) has been extensively utilized to
recover sentence and document-level encoding capability from pre-trained
language models. In this work, we question the length generalizability of
CL-based models, i.e., their vulnerability towards length-induced semantic
shift. We verify not only that length vulnerability is a significant yet
overlooked research gap, but we can devise unsupervised CL methods solely
depending on the semantic signal provided by document length. We first derive
the theoretical foundations underlying length attacks, showing that elongating
a document would intensify the high intra-document similarity that is already
brought by CL. Moreover, we found that isotropy promised by CL is highly
dependent on the length range of text exposed in training. Inspired by these
findings, we introduce a simple yet universal document representation learning
framework, LA(SER)$^{3}$: length-agnostic self-reference for semantically
robust sentence representation learning, achieving state-of-the-art
unsupervised performance on the standard information retrieval benchmark.

通过对文档长度的语义信号进行研究，本文提出了一种简单且通用的文档表示学习框架 LA (SER)$^{3}$，实现了在标准信息检索基准上的最先进的无监督性能。

文档级语义的长度既是诅咒也是福泽

Length is a Curse and a Blessing for Document-level Semantics

Existing NTMs with contrastive learning suffer from the sample bias problem
owing to the word frequency-based sampling strategy, which may result in false
negative samples with similar semantics to the prototypes. In this paper, we
aim to explore the efficient sampling strategy and contrastive learning in NTMs
to address the aforementioned issue. We propose a new sampling assumption that
negative samples should contain words that are semantically irrelevant to the
prototype. Based on it, we propose the graph contrastive topic model (GCTM),
which conducts graph contrastive learning (GCL) using informative positive and
negative samples that are generated by the graph-based sampling strategy
leveraging in-depth correlation and irrelevance among documents and words. In
GCTM, we first model the input document as the document word bipartite graph
(DWBG), and construct positive and negative word co-occurrence graphs (WCGs),
encoded by graph neural networks, to express in-depth semantic correlation and
irrelevance among words. Based on the DWBG and WCGs, we design the
document-word information propagation (DWIP) process to perform the edge
perturbation of DWBG, based on multi-hop correlations/irrelevance among
documents and words. This yields the desired negative and positive samples,
which will be utilized for GCL together with the prototypes to improve learning
document topic representations and latent topics. We further show that GCL can
be interpreted as the structured variational graph auto-encoder which maximizes
the mutual information of latent topic representations of different
perspectives on DWBG. Experiments on several benchmark datasets demonstrate the
effectiveness of our method for topic coherence and document representation
learning compared with existing SOTA methods.

本文提出了一种基于图的对比学习方法，使用生成的正负样本和原型来改善学习文档主题表示和潜在主题。通过对几个基准数据集的实验，证明了该方法在主题连贯性和文档表示学习方面的有效性。

图形对比主题模型

Graph Contrastive Topic Model

Relational understanding is critical for a number of visually-rich documents
(VRDs) understanding tasks. Through multi-modal pre-training, recent studies
provide comprehensive contextual representations and exploit them as prior
knowledge for downstream tasks. In spite of their impressive results, we
observe that the widespread relational hints (e.g., relation of key/value
fields on receipts) built upon contextual knowledge are not excavated yet. To
mitigate this gap, we propose DocReL, a Document Relational Representation
Learning framework. The major challenge of DocReL roots in the variety of
relations. From the simplest pairwise relation to the complex global structure,
it is infeasible to conduct supervised training due to the definition of
relation varies and even conflicts in different tasks. To deal with the
unpredictable definition of relations, we propose a novel contrastive learning
task named Relational Consistency Modeling (RCM), which harnesses the fact that
existing relations should be consistent in differently augmented positive
views. RCM provides relational representations which are more compatible to the
urgent need of downstream tasks, even without any knowledge about the exact
definition of relation. DocReL achieves better performance on a wide variety of
VRD relational understanding tasks, including table structure recognition, key
information extraction and reading order detection.

本研究提出了 DocReL，一种文本关系表示学习框架，其中包含一种新的对比学习任务 Relational Consistency Modeling（RCM），并利用了上下文信息来改进视觉丰富文档中关系理解，如表结构识别，关键信息提取和阅读顺序检测。