In recent years, the rise of generative self-supervised learning (SSL)
paradigms has exhibited impressive performance across visual, language, and
multi-modal domains. While the varied designs of generative SSL objectives lead
to distinct properties in downstream tasks, a theoretical understanding of
these differences remains largely unexplored. In this paper, we establish the
first theoretical comparisons between two leading generative SSL paradigms:
autoregressive SSL and masked SSL. Through establishing theoretical frameworks,
we elucidate the strengths and limitations of autoregressive and masked SSL
within the primary evaluation tasks of classification and content generation.
Our findings demonstrate that in classification tasks, the flexibility of
targeted tokens in masked SSL fosters more inter-sample connections compared to
the fixed position of target tokens in autoregressive SSL, which yields
superior clustering performance. In content generation tasks, the misalignment
between the flexible lengths of test samples and the fixed length of unmasked
texts in masked SSL (vs. flexible lengths of conditional texts in
autoregressive SSL) hinders its generation performance. To leverage each
other's strengths and mitigate weaknesses, we propose diversity-enhanced
autoregressive and variable-length masked objectives, which substantially
improve the classification performance of autoregressive SSL and the generation
performance of masked SSL. Code is available at
this https URL

通过建立理论框架，比较了两种主要生成式自监督学习范式 —— 自回归 SSL 和掩码 SSL 在分类和内容生成任务中的优势和局限性，并提出了多样性增强的自回归和可变长度掩码目标，显著提高了自回归 SSL 的分类性能和掩码 SSL 的生成性能。