Self-supervised representation learning often uses data augmentations to
induce some invariance to "style" attributes of the data. However, with
downstream tasks generally unknown at training time, it is difficult to deduce
a priori which attributes of the data are indeed "style" and can be safely
discarded. To address this, we introduce a more principled approach that seeks
to disentangle style features rather than discard them. The key idea is to add
multiple style embedding spaces where: (i) each is invariant to all-but-one
augmentation; and (ii) joint entropy is maximized. We formalize our structured
data-augmentation procedure from a causal latent-variable-model perspective,
and prove identifiability of both content and (multiple blocks of) style
variables. We empirically demonstrate the benefits of our approach on synthetic
datasets and then present promising but limited results on ImageNet.

自我监督表示学习经常使用数据增强来诱导对数据的 “风格” 属性的某种不变性。然而，由于在训练时通常不知道下游任务，很难事先推断哪些属性实际上是 “风格”，并且可以安全地丢弃。为了解决这个问题，我们引入了一种更有原则性的方法，旨在解开 “风格” 特征而不是丢弃它们。关键思想是添加多个风格嵌入空间，其中：（i）每个空间对除一个增强操作外都不变；（ii）联合熵被最大化。我们从因果潜变量模型的角度形式化了我们的结构化数据增强过程，并证明了内容和（多个模块的）风格变量的可辨识性。我们在合成数据集上通过实验证明了我们方法的好处，然后在 ImageNet 上展示了有限但有希望的结果。

自监督解缠：利用数据增强中的结构

Self-Supervised Disentanglement by Leveraging Structure in Data  Augmentations

Style is a significant component of natural language text, reflecting a
change in the tone of text while keeping the underlying information the same.
Even though programming languages have strict syntax rules, they also have
style. Code can be written with the same functionality but using different
language features. However, programming style is difficult to quantify, and
thus as part of this work, we define style attributes, specifically for Python.
To build a definition of style, we utilized hierarchical clustering to capture
a style definition without needing to specify transformations. In addition to
defining style, we explore the capability of a pre-trained code language model
to capture information about code style. To do this, we fine-tuned pre-trained
code-language models and evaluated their performance in code style transfer
tasks.

本研究利用 Python 编程语言作为样本，定义了编程代码的风格属性，并探讨了预训练代码语言模型对代码风格的识别能力。我们使用了分层聚类的方法来获取代码风格的定义，通过微调预训练模型，我们还评估了其在代码风格转换中的性能表现。