State-of-the-art computer vision models are mostly trained with supervised
learning using human-labeled images, which limits their scalability due to the
expensive annotation cost. While self-supervised representation learning has
achieved impressive progress, it still requires a second stage of finetuning on
labeled data. On the other hand, models pre-trained with large-scale text-image
supervision (e.g., CLIP) have enabled zero-shot transfer to downstream image
classification tasks. However, the zero-shot performance of CLIP-like models
are often insufficient for real-world adoption. In this paper, we aim to
leverage the abundant unlabeled data from a target domain to improve the
performance of a pre-trained zero-shot classifier, by unsupervised finetuning
of the pre-trained model. We propose Masked Unsupervised Self-Training (MUST),
a new unsupervised adaptation method which leverages two different and
complementary sources of training signals: pseudo-labels and raw images. MUST
jointly optimizes three objectives to learn both class-level global feature and
pixel-level local feature and enforces a regularization between the two. We
demonstrate the efficacy of MUST on a variety of downstream tasks, where it
improves upon CLIP by a large margin. MUST also outperforms supervised few-shot
adaptation methods. It achieves a top-1 accuracy of 77.7% on ImageNet using
ViT-B, +9.4% higher than CLIP, and +6.2% higher than 16-shot CLIP adaptation.
Our code is available at this https URL.

本文提出了一种利用目标领域丰富的无标注数据改进已预训练的零样本分类器的无监督微调方法 Masked Unsupervised Self-Training（MUST）并在各种下游任务上证明了其有效性，包括在 ImageNet 上使用 ViT-B 的准确率高达 77.7％，比 CLIP 高 9.4％，比 16-shot CLIP 适应高 6.2％。

无标签图像分类的屏蔽无监督自训练

Masked Unsupervised Self-training for Label-free Image Classification

This paper studies "unsupervised finetuning", the symmetrical problem of the
well-known "supervised finetuning". Given a pretrained model and small-scale
unlabeled target data, unsupervised finetuning is to adapt the representation
pretrained from the source domain to the target domain so that better transfer
performance can be obtained. This problem is more challenging than the
supervised counterpart, as the low data density in the small-scale target data
is not friendly for unsupervised learning, leading to the damage of the
pretrained representation and poor representation in the target domain. In this
paper, we find the source data is crucial when shifting the finetuning paradigm
from supervise to unsupervise, and propose two simple and effective strategies
to combine source and target data into unsupervised finetuning: "sparse source
data replaying", and "data mixing". The motivation of the former strategy is to
add a small portion of source data back to occupy their pretrained
representation space and help push the target data to reside in a smaller
compact space; and the motivation of the latter strategy is to increase the
data density and help learn more compact representation. To demonstrate the
effectiveness of our proposed ``unsupervised finetuning'' strategy, we conduct
extensive experiments on multiple different target datasets, which show better
transfer performance than the naive strategy.

本文研究了无监督微调的问题，提出了两种简单有效的策略来将源数据和目标数据进行组合以实现更好的传递性能。通过在多个不同的目标数据集上进行广泛的实验，表明了所提出的 “无监督微调” 策略比朴素策略具有更好的传递性能。

无监督微调

Unsupervised Finetuning

We present a learning based approach for multi-view stereopsis (MVS). While
current deep MVS methods achieve impressive results, they crucially rely on
ground-truth 3D training data, and acquisition of such precise 3D geometry for
supervision is a major hurdle. Our framework instead leverages photometric
consistency between multiple views as supervisory signal for learning depth
prediction in a wide baseline MVS setup. However, naively applying photo
consistency constraints is undesirable due to occlusion and lighting changes
across views. To overcome this, we propose a robust loss formulation that: a)
enforces first order consistency and b) for each point, selectively enforces
consistency with some views, thus implicitly handling occlusions. We
demonstrate our ability to learn MVS without 3D supervision using a real
dataset, and show that each component of our proposed robust loss results in a
significant improvement. We qualitatively observe that our reconstructions are
often more complete than the acquired ground truth, further showing the merits
of this approach. Lastly, our learned model generalizes to novel settings, and
our approach allows adaptation of existing CNNs to datasets without
ground-truth 3D by unsupervised finetuning. Project webpage:
this https URL

提出了一种基于学习的多视点立体感知方法，可以在宽基线 MVS 环境下学习深度预测并通过稳健的损失函数处理遮挡和光照等问题。该方法不需要使用三维监督数据进行训练，并且可以通过无监督的微调实现对新数据集的适应。