In this paper we delve into the properties of transformers, attained through
self-supervision, in the point cloud domain. Specifically, we evaluate the
effectiveness of Masked Autoencoding as a pretraining scheme, and explore
Momentum Contrast as an alternative. In our study we investigate the impact of
data quantity on the learned features, and uncover similarities in the
transformer's behavior across domains. Through comprehensive visualiations, we
observe that the transformer learns to attend to semantically meaningful
regions, indicating that pretraining leads to a better understanding of the
underlying geometry. Moreover, we examine the finetuning process and its effect
on the learned representations. Based on that, we devise an unfreezing strategy
which consistently outperforms our baseline without introducing any other
modifications to the model or the training pipeline, and achieve
state-of-the-art results in the classification task among transformer models.

本文研究了 transformers 在点云领域中通过自监督学习所获得的特性，探讨了其预训练方案的有效性，分析了数据数量对网络特征的影响，提出了一种解冻策略，并在分类任务中取得了最优结果。

可解释的自监督变压器深入探究点云

A deep dive into explainable self-supervised transformers for point  clouds

We present Momentum Contrast (MoCo) for unsupervised visual representation
learning. From a perspective on contrastive learning as dictionary look-up, we
build a dynamic dictionary with a queue and a moving-averaged encoder. This
enables building a large and consistent dictionary on-the-fly that facilitates
contrastive unsupervised learning. MoCo provides competitive results under the
common linear protocol on ImageNet classification. More importantly, the
representations learned by MoCo transfer well to downstream tasks. MoCo can
outperform its supervised pre-training counterpart in 7 detection/segmentation
tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large
margins. This suggests that the gap between unsupervised and supervised
representation learning has been largely closed in many vision tasks.

用于视觉表示学习的动量对比（MoCo）利用队列和移动平均编码器构建动态字典，并提供具有竞争力的结果并实现下游任务的迁移。