Self-supervised learning models have been shown to learn rich visual
representations without requiring human annotations. However, in many
real-world scenarios, labels are partially available, motivating a recent line
of work on semi-supervised methods inspired by self-supervised principles. In
this paper, we propose a conceptually simple yet empirically powerful approach
to turn clustering-based self-supervised methods such as SwAV or DINO into
semi-supervised learners. More precisely, we introduce a multi-task framework
merging a supervised objective using ground-truth labels and a self-supervised
objective relying on clustering assignments with a single cross-entropy loss.
This approach may be interpreted as imposing the cluster centroids to be class
prototypes. Despite its simplicity, we provide empirical evidence that our
approach is highly effective and achieves state-of-the-art performance on
CIFAR100 and ImageNet.

本文提出了一种将基于聚类的自监督方法转化为半监督学习的方法，通过引入一个多任务框架将监督学习和自监督学习结合，证明该方法在 CIFAR100 和 ImageNet 上的表现是最优的。

自监督聚类简化半监督学习

Semi-supervised learning made simple with self-supervised clustering

Unsupervised representation learning methods like SwAV are proved to be
effective in learning visual semantics of a target dataset. The main idea
behind these methods is that different views of a same image represent the same
semantics. In this paper, we further introduce an add-on module to facilitate
the injection of the knowledge accounting for spatial cross correlations among
the samples. This in turn results in distilling intra-class information
including feature level locations and cross similarities between same-class
instances. The proposed add-on can be added to existing methods such as the
SwAV. We can later remove the add-on module for inference without any
modification of the learned weights. Through an extensive set of empirical
evaluations, we verify that our method yields an improved performance in
detecting the class activation maps, top-1 classification accuracy, and
down-stream tasks such as object detection, with different configuration
settings.

该论文提出一个可添加到 SwAV 等现有方法中的附加模块，可以更好地学习图像语义中的空间交叉相关性和内部类信息，并提高物体检测等下游任务的性能。

空间交叉注意力提高自监督视觉表示学习

Spatial Cross-Attention Improves Self-Supervised Visual Representation Learning

Modern deep learning applications require increasingly more compute to train
state-of-the-art models. To address this demand, large corporations and
institutions use dedicated High-Performance Computing clusters, whose
construction and maintenance are both environmentally costly and well beyond
the budget of most organizations. As a result, some research directions become
the exclusive domain of a few large industrial and even fewer academic actors.
To alleviate this disparity, smaller groups may pool their computational
resources and run collaborative experiments that benefit all participants. This
paradigm, known as grid- or volunteer computing, has seen successful
applications in numerous scientific areas. However, using this approach for
machine learning is difficult due to high latency, asymmetric bandwidth, and
several challenges unique to volunteer computing. In this work, we carefully
analyze these constraints and propose a novel algorithmic framework designed
specifically for collaborative training. We demonstrate the effectiveness of
our approach for SwAV and ALBERT pretraining in realistic conditions and
achieve performance comparable to traditional setups at a fraction of the cost.
Finally, we provide a detailed report of successful collaborative language
model pretraining with 40 participants.

本文研究了 Deep Learning 中在协作训练上的瓶颈，提出了一种适用于协作训练的新算法框架，并在实际条件下展示了该方法对 SwAV 和 ALBERT 的预训练效果，结果表明该方法的性能与传统设置相当，成本仅为传统设置的一小部分。最后，提供了一个由 40 名参与者组成的成功的协作语言模型预训练的详细报告。

开放协作中的分布式深度学习

Distributed Deep Learning in Open Collaborations

Unsupervised image representations have significantly reduced the gap with
supervised pretraining, notably with the recent achievements of contrastive
learning methods. These contrastive methods typically work online and rely on a
large number of explicit pairwise feature comparisons, which is computationally
challenging. In this paper, we propose an online algorithm, SwAV, that takes
advantage of contrastive methods without requiring to compute pairwise
comparisons. Specifically, our method simultaneously clusters the data while
enforcing consistency between cluster assignments produced for different
augmentations (or views) of the same image, instead of comparing features
directly as in contrastive learning. Simply put, we use a swapped prediction
mechanism where we predict the cluster assignment of a view from the
representation of another view. Our method can be trained with large and small
batches and can scale to unlimited amounts of data. Compared to previous
contrastive methods, our method is more memory efficient since it does not
require a large memory bank or a special momentum network. In addition, we also
propose a new data augmentation strategy, multi-crop, that uses a mix of views
with different resolutions in place of two full-resolution views, without
increasing the memory or compute requirements much. We validate our findings by
achieving 75.3% top-1 accuracy on ImageNet with ResNet-50, as well as
surpassing supervised pretraining on all the considered transfer tasks.

本文介绍了一种在线聚类算法 SwAV，在不需要计算成对对比的基础上，通过使用一种被称为预测机制的方式来建立视图之间的一致性，并提出了一种新的数据增强策略多裁剪，在 ImageNet 数据集上达到 75.3％的 top-1 准确率，并超出了所有考虑的迁移任务的监督预训练。