Good data augmentation is one of the key factors that lead to the empirical
success of self-supervised representation learning such as contrastive learning
and masked language modeling, yet theoretical understanding of its role in
learning good representations remains limited. Recent work has built the
connection between self-supervised learning and approximating the top
eigenspace of a graph Laplacian operator. Learning a linear probe on top of
such features can naturally be connected to RKHS regression. In this work, we
use this insight to perform a statistical analysis of augmentation-based
pretraining. We start from the isometry property, a key geometric
characterization of the target function given by the augmentation. Our first
main theorem provides, for an arbitrary encoder, near tight bounds for both the
estimation error incurred by fitting the linear probe on top of the encoder,
and the approximation error entailed by the fitness of the RKHS the encoder
learns. Our second main theorem specifically addresses the case where the
encoder extracts the top-d eigenspace of a Monte-Carlo approximation of the
underlying kernel with the finite pretraining samples. Our analysis completely
disentangles the effects of the model and the augmentation. A key ingredient in
our analysis is the augmentation complexity, which we use to quantitatively
compare different augmentations and analyze their impact on downstream
performance on synthetic and real datasets.

本文从 data augmentation 在 self-supervised learning 中的作用入手，结合 RKHS regression 和 Monte-Carlo approximation ，分析了不同 data augmentation 对模型性能的影响。