Self-Supervised Learning (SSL) is a valuable and robust training methodology
for contemporary Deep Neural Networks (DNNs), enabling unsupervised pretraining
on a `pretext task' that does not require ground-truth labels/annotation. This
allows efficient representation learning from massive amounts of unlabeled
training data, which in turn leads to increased accuracy in a `downstream task'
by exploiting supervised transfer learning. Despite the relatively
straightforward conceptualization and applicability of SSL, it is not always
feasible to collect and/or to utilize very large pretraining datasets,
especially when it comes to real-world application settings. In particular, in
cases of specialized and domain-specific application scenarios, it may not be
achievable or practical to assemble a relevant image pretraining dataset in the
order of millions of instances or it could be computationally infeasible to
pretrain at this scale. This motivates an investigation on the effectiveness of
common SSL pretext tasks, when the pretraining dataset is of relatively
limited/constrained size. In this context, this work introduces a taxonomy of
modern visual SSL methods, accompanied by detailed explanations and insights
regarding the main categories of approaches, and, subsequently, conducts a
thorough comparative experimental evaluation in the low-data regime, targeting
to identify: a) what is learnt via low-data SSL pretraining, and b) how do
different SSL categories behave in such training scenarios. Interestingly, for
domain-specific downstream tasks, in-domain low-data SSL pretraining
outperforms the common approach of large-scale pretraining on general datasets.
Grounded on the obtained results, valuable insights are highlighted regarding
the performance of each category of SSL methods, which in turn suggest
straightforward future research directions in the field.

在相对有限 / 受约束的数据集中，通过比较实验评估了现代视觉自监督学习方法的有效性，发现针对特定领域的下游任务，在领域内的低数据量自监督预训练优于在通用数据集上进行大规模预训练，为该领域中各类自监督学习方法的性能提供了有价值的见解，并提出了直接的未来研究方向。