Contrastive unsupervised representation learning (CURL) is the
state-of-the-art technique to learn representations (as a set of features) from
unlabelled data. While CURL has collected several empirical successes recently,
theoretical understanding of its performance was still missing. In a recent
work, Arora et al. (2019) provide the first generalisation bounds for CURL,
relying on a Rademacher complexity. We extend their framework to the flexible
PAC-Bayes setting, allowing us to deal with the non-iid setting. We present
PAC-Bayesian generalisation bounds for CURL, which are then used to derive a
new representation learning algorithm. Numerical experiments on real-life
datasets illustrate that our algorithm achieves competitive accuracy, and
yields non-vacuous generalisation bounds.

本文介绍了对无标签数据进行特征提取的一种技术 Contrastive unsupervised representation learning (CURL)，论文扩展了 Arora 等人 2019 年的工作，提出了 PAC-Bayesian generalization bounds 来处理非 iid 情况，并制定了一种新的表示学习算法，实验结果表明此算法具有不俗的精度和泛化性能。

PAC-Bayesian 无对照非监督表示学习

PAC-Bayesian Contrastive Unsupervised Representation Learning

Effective modeling of electronic health records (EHR) is rapidly becoming an
important topic in both academia and industry. A recent study showed that using
the graphical structure underlying EHR data (e.g. relationship between
diagnoses and treatments) improves the performance of prediction tasks such as
heart failure prediction. However, EHR data do not always contain complete
structure information. Moreover, when it comes to claims data, structure
information is completely unavailable to begin with. Under such circumstances,
can we still do better than just treating EHR data as a flat-structured
bag-of-features? In this paper, we study the possibility of jointly learning
the hidden structure of EHR while performing supervised prediction tasks on EHR
data. Specifically, we discuss that Transformer is a suitable basis model to
learn the hidden EHR structure, and propose Graph Convolutional Transformer,
which uses data statistics to guide the structure learning process. The
proposed model consistently outperformed previous approaches empirically, on
both synthetic data and publicly available EHR data, for various prediction
tasks such as graph reconstruction and readmission prediction, indicating that
it can serve as an effective general-purpose representation learning algorithm
for EHR data.

本文研究如何在缺少信息的情况下，利用 Transformer 模型来学习电子健康记录的隐含结构，并提出了一种使用数据统计来指导结构学习过程的图卷积 Transformer 模型。实验表明，该模型在预测任务中比以往方法表现更好，可作为一种有效的 EHR 数据表示学习算法。