Given the claims of improved text generation quality across various
pre-trained neural models, we consider the coherence evaluation of machine
generated text to be one of the principal applications of coherence models that
needs to be investigated. Prior work in neural coherence modeling has primarily
focused on devising new architectures for solving the permuted document task.
We instead use a basic model architecture and show significant improvements
over state of the art within the same training regime. We then design a harder
self-supervision objective by increasing the ratio of negative samples within a
contrastive learning setup, and enhance the model further through automatic
hard negative mining coupled with a large global negative queue encoded by a
momentum encoder. We show empirically that increasing the density of negative
samples improves the basic model, and using a global negative queue further
improves and stabilizes the model while training with hard negative samples. We
evaluate the coherence model on task-independent test sets that resemble
real-world applications and show significant improvements in coherence
evaluations of downstream tasks.

本文研究利用基本的神经模型架构，结合对比学习中的负采样和硬负采样策略以及大规模全局负采样队列的增强效应，在任务独立测试集上进行一致性模型的评估和下游任务的改进。结果表明，增加负样本密度和使用全局负采样队列结合硬负采样可以显著地提高模型的性能表现。

重新思考自监督目标，用于可泛化的连贯建模

Rethinking Self-Supervision Objectives for Generalizable Coherence  Modeling

Recently, heterogeneous Graph Neural Networks (GNNs) have become a de facto
model for analyzing HGs, while most of them rely on a relative large number of
labeled data. In this work, we investigate Contrastive Learning (CL), a key
component in self-supervised approaches, on HGs to alleviate the label scarcity
problem. We first generate multiple semantic views according to metapaths and
network schemas. Then, by pushing node embeddings corresponding to different
semantic views close to each other (positives) and pulling other embeddings
apart (negatives), one can obtain informative representations without human
annotations. However, this CL approach ignores the relative hardness of
negative samples, which may lead to suboptimal performance. Considering the
complex graph structure and the smoothing nature of GNNs, we propose a
structure-aware hard negative mining scheme that measures hardness by
structural characteristics for HGs. By synthesizing more negative nodes, we
give larger weights to harder negatives with limited computational overhead to
further boost the performance. Empirical studies on three real-world datasets
show the effectiveness of our proposed method. The proposed method consistently
outperforms existing state-of-the-art methods and notably, even surpasses
several supervised counterparts.

本文通过结构感知的硬负样本挖掘方案提出了一个基于自监督学习的对比学习方法，可以在少量标记数据的情况下提高异构图分析模型的准确性，并通过三个真实数据集的实证研究验证了该方法的有效性。

异构图对比学习中的结构感知难例挖掘

Structure-Aware Hard Negative Mining for Heterogeneous Graph Contrastive  Learning

Self-supervised learning holds promise in leveraging large amounts of
unlabeled data, however much of its progress has thus far been limited to
highly curated pre-training data such as ImageNet. We explore the effects of
contrastive learning from larger, less-curated image datasets such as YFCC, and
find there is indeed a large difference in the resulting representation
quality. We hypothesize that this curation gap is due to a shift in the
distribution of image classes -- which is more diverse and heavy-tailed --
resulting in less relevant negative samples to learn from. We test this
hypothesis with a new approach, Divide and Contrast (DnC), which alternates
between contrastive learning and clustering-based hard negative mining. When
pretrained on less curated datasets, DnC greatly improves the performance of
self-supervised learning on downstream tasks, while remaining competitive with
the current state-of-the-art on curated datasets.

本文研究自监督学习在大规模数据集上的应用，提出了一种基于对比学习与聚类的硬负样本挖掘方法（DnC），在 less-curated 数据集上的预训练，可以显著提高自监督学习在后续任务上的表现效果，并与目前在高度筛选数据集上的最新水平保持竞争力。